For autonomous vehicles and driver assist systems,path planning and collision avoidance algorithms benefit from accurate predictions of future location of other vehicles and intent of their drivers.In the literature,t...For autonomous vehicles and driver assist systems,path planning and collision avoidance algorithms benefit from accurate predictions of future location of other vehicles and intent of their drivers.In the literature,the algorithms that provide driver intent belong to two categories:those that use physics based models with some type of filtering,and machine learning based approaches.In this paper we employ barrier functions(BF)to decide driver intent.BFs are typically used to prove safety by establishing forward invariance of an admissible set.Here,we decide if the“target”vehicle is violating one or more possibly fictitious(i.e.,non-physical)barrier constraints determined based on the context provided by the road geometry.The algorithm has a very small computational footprint and better false positive and negative rates than some of the alternatives.The predicted intent is then used by a control barrier function(CBF)based collision avoidance system to prevent unnecessary interventions,for either an autonomous or human-driven vehicle.展开更多
Despite the gradual transformation of traditional manufacturing by the Human-Robot Collaboration Assembly(HRCA),challenges remain in the robot’s ability to understand and predict human assembly intentions.This study ...Despite the gradual transformation of traditional manufacturing by the Human-Robot Collaboration Assembly(HRCA),challenges remain in the robot’s ability to understand and predict human assembly intentions.This study aims to enhance the robot’s comprehension and prediction capabilities of operator assembly intentions by capturing and analyzing operator behavior and movements.We propose a video feature extraction method based on the Temporal Shift Module Network(TSM-ResNet50)to extract spatiotemporal features from assembly videos and differentiate various assembly actions using feature differences between video frames.Furthermore,we construct an action recognition and segmentation model based on the Refined-Multi-Scale Temporal Convolutional Network(Refined-MS-TCN)to identify assembly action intervals and accurately acquire action categories.Experiments on our self-built reducer assembly action dataset demonstrate that our network can classify assembly actions frame by frame,achieving an accuracy rate of 83%.Additionally,we develop a HiddenMarkovModel(HMM)integrated with assembly task constraints to predict operator assembly intentions based on the probability transition matrix and assembly task constraints.The experimental results show that our method for predicting operator assembly intentions can achieve an accuracy of 90.6%,which is a 13.3%improvement over the HMM without task constraints.展开更多
High complexity and uncertainty of air combat pose significant challenges to target intention prediction.Current interpolation methods for data pre-processing and wrangling have limitations in capturing interrelations...High complexity and uncertainty of air combat pose significant challenges to target intention prediction.Current interpolation methods for data pre-processing and wrangling have limitations in capturing interrelationships among intricate variable patterns.Accordingly,this study proposes a Mogrifier gate recurrent unit-D(Mog-GRU-D)model to address the com-bat target intention prediction issue under the incomplete infor-mation condition.The proposed model directly processes miss-ing data while reducing the independence between inputs and output states.A total of 1200 samples from twelve continuous moments are captured through the combat simulation system,each of which consists of seven dimensional features.To bench-mark the experiment,a missing valued dataset has been gener-ated by randomly removing 20%of the original data.Extensive experiments demonstrate that the proposed model obtains the state-of-the-art performance with an accuracy of 73.25%when dealing with incomplete information.This study provides possi-ble interpretations for the principle of target interactive mecha-nism,highlighting the model’s effectiveness in potential air war-fare implementation.展开更多
Slot filling and intent prediction are basic tasks in capturing semantic frame of human utterances.Slots and intent have strong correlation for semantic frame parsing.For each utterance,a specific intent type is gener...Slot filling and intent prediction are basic tasks in capturing semantic frame of human utterances.Slots and intent have strong correlation for semantic frame parsing.For each utterance,a specific intent type is generally determined with the indication information of words having slot tags(called as slot words),and in reverse the intent type decides that words of certain categories should be used to fill as slots.However,the Intent-Slot correlation is rarely modeled explicitly in existing studies,and hence may be not fully exploited.In this paper,we model Intent-Slot correlation explicitly and propose a new framework for joint intent prediction and slot filling.Firstly,we explore the effects of slot words on intent by differentiating them from the other words,and we recognize slot words by solving a sequence labeling task with the bi-directional long short-term memory(BiLSTM)model.Then,slot recognition information is introduced into attention-based intent prediction and slot filling to improve semantic results.In addition,we integrate the Slot-Gated mechanism into slot filling to model dependency of slots on intent.Finally,we obtain slot recognition,intent prediction and slot filling by training with joint optimization.Experimental results on the benchmark Air-line Travel Information System(ATIS)and Snips datasets show that our Intent-Slot correlation model achieves state-of-the-art semantic frame performance with a lightweight structure.展开更多
Recently,wearable gait-assist robots have been evolving towards using soft materials designed for the elderly rather than individuals with disabilities,which emphasize modularization,simplification,and weight reductio...Recently,wearable gait-assist robots have been evolving towards using soft materials designed for the elderly rather than individuals with disabilities,which emphasize modularization,simplification,and weight reduction.Thus,synchronizing the robotic assistive force with that of the user’s leg movements is crucial for usability,which requires accurate recognition of the user’s gait intent.In this study,we propose a deep learning model capable of identifying not only gait mode and gait phase but also phase progression.Utilizing data from five inertial measurement units placed on the body,the proposed two-stage architecture incorporates a bidirectional long short-term memory-based model for robust classification of locomotion modes and phases.Subsequently,phase progression is estimated through 1D convolutional neural network-based regressors,each dedicated to a specific phase.The model was evaluated on a diverse dataset encompassing level walking,stair ascent and descent,and sit-to-stand activities from 10 healthy participants.The results demonstrate its ability to accurately classify locomotion phases and estimate phase progression.Accurate phase progression estimation is essential due to the age-related variability in gait phase durations,particularly evident in older adults,the primary demographic for gait-assist robots.These findings underscore the potential to enhance the assistance,comfort,and safety provided by gait-assist robots.展开更多
Driving intention prediction from a bird’s-eye view has always been an active research area. However,existing research, on one hand, has only focused on predicting lane change intention in highway scenarios and, on t...Driving intention prediction from a bird’s-eye view has always been an active research area. However,existing research, on one hand, has only focused on predicting lane change intention in highway scenarios and, on the other hand, has not modeled the influence and spatiotemporal relationship of surrounding vehicles. This study extends the application scenarios to urban road scenarios. A spatial-temporal convolutional long short-term memory(ConvLSTM) model is proposed to predict the vehicle’s lateral and longitudinal driving intentions simultaneously. This network includes two modules: the first module mines the information of the target vehicle using the long short-term memory(LSTM) network and the second module uses ConvLSTM to capture the spatial interactions and temporal evolution of surrounding vehicles simultaneously when modeling the influence of surrounding vehicles. The model is trained and verified on a real road dataset, and the results show that the spatial-temporal ConvLSTM model is superior to the traditional LSTM in terms of accuracy, precision, and recall, which helps improve the prediction accuracy at different time horizons.展开更多
Realizing autonomy is a hot research topic for automatic vehicles in recent years. For a long time, most of the efforts to this goal concentrate on understanding the scenes surrounding the ego-vehicle（autonomous vehi...Realizing autonomy is a hot research topic for automatic vehicles in recent years. For a long time, most of the efforts to this goal concentrate on understanding the scenes surrounding the ego-vehicle（autonomous vehicle itself）. By completing lowlevel vision tasks, such as detection, tracking and segmentation of the surrounding traffic participants, e.g., pedestrian, cyclists and vehicles, the scenes can be interpreted. However, for an autonomous vehicle, low-level vision tasks are largely insufficient to give help to comprehensive scene understanding. What are and how about the past, the on-going and the future of the scene participants？ This deep question actually steers the vehicles towards truly full automation, just like human beings. Based on this thoughtfulness, this paper attempts to investigate the interpretation of traffic scene in autonomous driving from an event reasoning view. To reach this goal, we study the most relevant literatures and the state-of-the-arts on scene representation, event detection and intention prediction in autonomous driving. In addition, we also discuss the open challenges and problems in this field and endeavor to provide possible solutions.展开更多
The state-of-the-art technology in the field of vehicle automation will lead to a mixed traffic environment in the coming years,where connected and automated vehicles have to interact with human-driven vehicles.In thi...The state-of-the-art technology in the field of vehicle automation will lead to a mixed traffic environment in the coming years,where connected and automated vehicles have to interact with human-driven vehicles.In this context,it is necessary to have intention prediction models with the capability of forecasting how the traffic scenario is going to evolve with respect to the physical state of vehicles,the possible maneuvers and the interactions between traffic participants within the seconds to come.This article presents a Bayesian approach for vehicle intention forecasting,utilizing a game-theoretic framework in the form of a Mixed Strategy Nash Equilibrium(MSNE)as a prior estimate to model the reciprocal influence between traffic participants.The likelihood is then computed based on the Kullback-Leibler divergence.The game is modeled as a static nonzero-sum polymatrix game with individual preferences,a well known strategic game.Finding the MSNE for these games is in the PPAD∩PLS complexity class,with polynomial-time tractability.The approach shows good results in simulations in the long term horizon(10s),with its computational complexity allowing for online applications.展开更多
In real-world driving scenarios,understanding the intentions of pedestrians in real-time is critical for the built environment safety when operating intelligent vehicles on the roads.Pedestrians crossing the street is...In real-world driving scenarios,understanding the intentions of pedestrians in real-time is critical for the built environment safety when operating intelligent vehicles on the roads.Pedestrians crossing the street is a common behavior that can easily lead to accidents.This paper presents a comprehensive review of the prediction of pedestrian crossing intentions,focusing on data,model structure,data representation,information extraction,prediction function,and associated models and challenges.The review highlights that data types,model generalization ability,and prediction uncertainty are key challenges on pedestrian crossing intention prediction.It identifies open challenges and opportunities for future research in pedestrian crossing intention prediction.展开更多
Trajectory prediction is an essential component in autonomous driving systems,as it can forecast the future movements of surrounding vehicles,thereby enhancing the decision-making and planning capabilities of autonomo...Trajectory prediction is an essential component in autonomous driving systems,as it can forecast the future movements of surrounding vehicles,thereby enhancing the decision-making and planning capabilities of autonomous driving systems.Traditional models relying on constant acceleration and constant velocity often experience a reduction in prediction accu-racy as the forecasted timeframe extends.This limitation makes it challenging to meet the demands for medium to long-term trajectory prediction.Conversely,data-driven models,particularly those based on Long Short-Term Memory(LSTM)neural networks,have demonstrated superior performance in medium to long-term trajectory prediction.Therefore,this study introduces a hierarchical LSTM-based method for vehicle trajectory prediction.Considering the difficulty of using a single LSTM model to predict trajectories for all driving intentions,the trajectory prediction task is decomposed into three sequential steps:driving intention prediction,lane change time prediction,and trajectory prediction.Furthermore,given that the driving intent and trajectory of a vehicle are always subject to the influence of the surrounding traffic flow,the predictive model proposed in this paper incorporates the interactional information of neighboring vehicle movements into the model input.The proposed method is trained and validated on the real vehicle trajectory dataset Next Generation Simulation.The results show that the proposed hierarchical LSTM method has a lower prediction error compared to the integral LSTM model.展开更多
文摘For autonomous vehicles and driver assist systems,path planning and collision avoidance algorithms benefit from accurate predictions of future location of other vehicles and intent of their drivers.In the literature,the algorithms that provide driver intent belong to two categories:those that use physics based models with some type of filtering,and machine learning based approaches.In this paper we employ barrier functions(BF)to decide driver intent.BFs are typically used to prove safety by establishing forward invariance of an admissible set.Here,we decide if the“target”vehicle is violating one or more possibly fictitious(i.e.,non-physical)barrier constraints determined based on the context provided by the road geometry.The algorithm has a very small computational footprint and better false positive and negative rates than some of the alternatives.The predicted intent is then used by a control barrier function(CBF)based collision avoidance system to prevent unnecessary interventions,for either an autonomous or human-driven vehicle.
文摘Despite the gradual transformation of traditional manufacturing by the Human-Robot Collaboration Assembly(HRCA),challenges remain in the robot’s ability to understand and predict human assembly intentions.This study aims to enhance the robot’s comprehension and prediction capabilities of operator assembly intentions by capturing and analyzing operator behavior and movements.We propose a video feature extraction method based on the Temporal Shift Module Network(TSM-ResNet50)to extract spatiotemporal features from assembly videos and differentiate various assembly actions using feature differences between video frames.Furthermore,we construct an action recognition and segmentation model based on the Refined-Multi-Scale Temporal Convolutional Network(Refined-MS-TCN)to identify assembly action intervals and accurately acquire action categories.Experiments on our self-built reducer assembly action dataset demonstrate that our network can classify assembly actions frame by frame,achieving an accuracy rate of 83%.Additionally,we develop a HiddenMarkovModel(HMM)integrated with assembly task constraints to predict operator assembly intentions based on the probability transition matrix and assembly task constraints.The experimental results show that our method for predicting operator assembly intentions can achieve an accuracy of 90.6%,which is a 13.3%improvement over the HMM without task constraints.
基金supported by the Aeronautical Science Foundation of China(2020Z023053002).
文摘High complexity and uncertainty of air combat pose significant challenges to target intention prediction.Current interpolation methods for data pre-processing and wrangling have limitations in capturing interrelationships among intricate variable patterns.Accordingly,this study proposes a Mogrifier gate recurrent unit-D(Mog-GRU-D)model to address the com-bat target intention prediction issue under the incomplete infor-mation condition.The proposed model directly processes miss-ing data while reducing the independence between inputs and output states.A total of 1200 samples from twelve continuous moments are captured through the combat simulation system,each of which consists of seven dimensional features.To bench-mark the experiment,a missing valued dataset has been gener-ated by randomly removing 20%of the original data.Extensive experiments demonstrate that the proposed model obtains the state-of-the-art performance with an accuracy of 73.25%when dealing with incomplete information.This study provides possi-ble interpretations for the principle of target interactive mecha-nism,highlighting the model’s effectiveness in potential air war-fare implementation.
文摘Slot filling and intent prediction are basic tasks in capturing semantic frame of human utterances.Slots and intent have strong correlation for semantic frame parsing.For each utterance,a specific intent type is generally determined with the indication information of words having slot tags(called as slot words),and in reverse the intent type decides that words of certain categories should be used to fill as slots.However,the Intent-Slot correlation is rarely modeled explicitly in existing studies,and hence may be not fully exploited.In this paper,we model Intent-Slot correlation explicitly and propose a new framework for joint intent prediction and slot filling.Firstly,we explore the effects of slot words on intent by differentiating them from the other words,and we recognize slot words by solving a sequence labeling task with the bi-directional long short-term memory(BiLSTM)model.Then,slot recognition information is introduced into attention-based intent prediction and slot filling to improve semantic results.In addition,we integrate the Slot-Gated mechanism into slot filling to model dependency of slots on intent.Finally,we obtain slot recognition,intent prediction and slot filling by training with joint optimization.Experimental results on the benchmark Air-line Travel Information System(ATIS)and Snips datasets show that our Intent-Slot correlation model achieves state-of-the-art semantic frame performance with a lightweight structure.
基金supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute(KHIDI)funded by the Ministry of Health&Welfare,Republic of Korea(Grant Number:RS-2022-KH129263).
文摘Recently,wearable gait-assist robots have been evolving towards using soft materials designed for the elderly rather than individuals with disabilities,which emphasize modularization,simplification,and weight reduction.Thus,synchronizing the robotic assistive force with that of the user’s leg movements is crucial for usability,which requires accurate recognition of the user’s gait intent.In this study,we propose a deep learning model capable of identifying not only gait mode and gait phase but also phase progression.Utilizing data from five inertial measurement units placed on the body,the proposed two-stage architecture incorporates a bidirectional long short-term memory-based model for robust classification of locomotion modes and phases.Subsequently,phase progression is estimated through 1D convolutional neural network-based regressors,each dedicated to a specific phase.The model was evaluated on a diverse dataset encompassing level walking,stair ascent and descent,and sit-to-stand activities from 10 healthy participants.The results demonstrate its ability to accurately classify locomotion phases and estimate phase progression.Accurate phase progression estimation is essential due to the age-related variability in gait phase durations,particularly evident in older adults,the primary demographic for gait-assist robots.These findings underscore the potential to enhance the assistance,comfort,and safety provided by gait-assist robots.
基金supported by the National Key Research and Development Program of China (No. 2017YFB0102601)the National Natural Science Foundation of China (No. 71671100)the Key Laboratory of Road and Traffic Engineering of the Ministry of Education, Tongji University。
文摘Driving intention prediction from a bird’s-eye view has always been an active research area. However,existing research, on one hand, has only focused on predicting lane change intention in highway scenarios and, on the other hand, has not modeled the influence and spatiotemporal relationship of surrounding vehicles. This study extends the application scenarios to urban road scenarios. A spatial-temporal convolutional long short-term memory(ConvLSTM) model is proposed to predict the vehicle’s lateral and longitudinal driving intentions simultaneously. This network includes two modules: the first module mines the information of the target vehicle using the long short-term memory(LSTM) network and the second module uses ConvLSTM to capture the spatial interactions and temporal evolution of surrounding vehicles simultaneously when modeling the influence of surrounding vehicles. The model is trained and verified on a real road dataset, and the results show that the spatial-temporal ConvLSTM model is superior to the traditional LSTM in terms of accuracy, precision, and recall, which helps improve the prediction accuracy at different time horizons.
基金supported by National Key R&D Program Project of China(No.2016YFB1001004)National Natural Science Foundation of China(Nos.61751308,61603057,61773311)+1 种基金China Postdoctoral Science Foundation(No.2017M613152)Collaborative Research with MSRA
文摘Realizing autonomy is a hot research topic for automatic vehicles in recent years. For a long time, most of the efforts to this goal concentrate on understanding the scenes surrounding the ego-vehicle（autonomous vehicle itself）. By completing lowlevel vision tasks, such as detection, tracking and segmentation of the surrounding traffic participants, e.g., pedestrian, cyclists and vehicles, the scenes can be interpreted. However, for an autonomous vehicle, low-level vision tasks are largely insufficient to give help to comprehensive scene understanding. What are and how about the past, the on-going and the future of the scene participants？ This deep question actually steers the vehicles towards truly full automation, just like human beings. Based on this thoughtfulness, this paper attempts to investigate the interpretation of traffic scene in autonomous driving from an event reasoning view. To reach this goal, we study the most relevant literatures and the state-of-the-arts on scene representation, event detection and intention prediction in autonomous driving. In addition, we also discuss the open challenges and problems in this field and endeavor to provide possible solutions.
文摘The state-of-the-art technology in the field of vehicle automation will lead to a mixed traffic environment in the coming years,where connected and automated vehicles have to interact with human-driven vehicles.In this context,it is necessary to have intention prediction models with the capability of forecasting how the traffic scenario is going to evolve with respect to the physical state of vehicles,the possible maneuvers and the interactions between traffic participants within the seconds to come.This article presents a Bayesian approach for vehicle intention forecasting,utilizing a game-theoretic framework in the form of a Mixed Strategy Nash Equilibrium(MSNE)as a prior estimate to model the reciprocal influence between traffic participants.The likelihood is then computed based on the Kullback-Leibler divergence.The game is modeled as a static nonzero-sum polymatrix game with individual preferences,a well known strategic game.Finding the MSNE for these games is in the PPAD∩PLS complexity class,with polynomial-time tractability.The approach shows good results in simulations in the long term horizon(10s),with its computational complexity allowing for online applications.
文摘In real-world driving scenarios,understanding the intentions of pedestrians in real-time is critical for the built environment safety when operating intelligent vehicles on the roads.Pedestrians crossing the street is a common behavior that can easily lead to accidents.This paper presents a comprehensive review of the prediction of pedestrian crossing intentions,focusing on data,model structure,data representation,information extraction,prediction function,and associated models and challenges.The review highlights that data types,model generalization ability,and prediction uncertainty are key challenges on pedestrian crossing intention prediction.It identifies open challenges and opportunities for future research in pedestrian crossing intention prediction.
基金supported by the Jilin Province Science and Technology Development Program(20210301023GX).
文摘Trajectory prediction is an essential component in autonomous driving systems,as it can forecast the future movements of surrounding vehicles,thereby enhancing the decision-making and planning capabilities of autonomous driving systems.Traditional models relying on constant acceleration and constant velocity often experience a reduction in prediction accu-racy as the forecasted timeframe extends.This limitation makes it challenging to meet the demands for medium to long-term trajectory prediction.Conversely,data-driven models,particularly those based on Long Short-Term Memory(LSTM)neural networks,have demonstrated superior performance in medium to long-term trajectory prediction.Therefore,this study introduces a hierarchical LSTM-based method for vehicle trajectory prediction.Considering the difficulty of using a single LSTM model to predict trajectories for all driving intentions,the trajectory prediction task is decomposed into three sequential steps:driving intention prediction,lane change time prediction,and trajectory prediction.Furthermore,given that the driving intent and trajectory of a vehicle are always subject to the influence of the surrounding traffic flow,the predictive model proposed in this paper incorporates the interactional information of neighboring vehicle movements into the model input.The proposed method is trained and validated on the real vehicle trajectory dataset Next Generation Simulation.The results show that the proposed hierarchical LSTM method has a lower prediction error compared to the integral LSTM model.
