Stereoscopic and multiview rendering are used for virtual reality and the synthetic generation of light fields from three-dimensional scenes.Because rendering multiple views using ray tracing techniques is computation...Stereoscopic and multiview rendering are used for virtual reality and the synthetic generation of light fields from three-dimensional scenes.Because rendering multiple views using ray tracing techniques is computationally expensive,the utilization of multiprocessor machines is necessary to achieve real-time frame rates.In this study,we propose a dynamic load-balancing algorithm for real-time multiview path tracing on multi-compute device platforms.The proposed algorithm was adapted to heterogeneous hardware combinations and dynamic scenes in real time.We show that on a heterogeneous dual-GPU platform,our implementation reduces the rendering time by an average of approximately 30%–50%compared with that of a uniform workload distribution,depending on the scene and number of views.展开更多
Traffic accidents are caused by driver fatigue or distraction in many cases.To prevent accidents,several low-cost hypovigilance(hypo-V)systems were developed in the past based on a multimodal-hybrid(physiological and ...Traffic accidents are caused by driver fatigue or distraction in many cases.To prevent accidents,several low-cost hypovigilance(hypo-V)systems were developed in the past based on a multimodal-hybrid(physiological and behavioral)feature set.Similarly in this paper,real-time driver inattention and fatigue(Hypo-Driver)detection system is proposed through multi-view cameras and biosignal sensors to extract hybrid features.The considered features are derived from non-intrusive sensors that are related to the changes in driving behavior and visual facial expressions.To get enhanced visual facial features in uncontrolled environment,three cameras are deployed on multiview points(0◦,45◦,and 90◦)of the drivers.To develop a Hypo-Driver system,the physiological signals(electroencephalography(EEG),electrocardiography(ECG),electro-myography(sEMG),and electrooculography(EOG))and behavioral information(PERCLOS70-80-90%,mouth aspect ratio(MAR),eye aspect ratio(EAR),blinking frequency(BF),head-titled ratio(HT-R))are collected and pre-processed,then followed by feature selection and fusion techniques.The driver behaviors are classified into five stages such as normal,fatigue,visual inattention,cognitive inattention,and drowsy.This improved hypo-Driver system utilized trained behavioral features by a convolutional neural network(CNNs),recurrent neural network and long short-term memory(RNN-LSTM)model is used to extract physiological features.After fusion of these features,the Hypo-Driver system is classified hypo-V into five stages based on trained layers and dropout-layer in the deep-residual neural network(DRNN)model.To test the performance of a hypo-Driver system,data from 20 drivers are acquired.The results of Hypo-Driver compared to state-of-theart methods are presented.Compared to the state-of-the-art Hypo-V system,on average,the Hypo-Driver system achieved a detection accuracy(AC)of 96.5%.The obtained results indicate that the Hypo-Driver system based on multimodal and multiview features outperforms other state-of-the-art driver Hypo-V systems by handling many anomalies.展开更多
Human activity recognition is a recent area of research for researchers.Activity recognition has many applications in smart homes to observe and track toddlers or oldsters for their safety,monitor indoor and outdoor a...Human activity recognition is a recent area of research for researchers.Activity recognition has many applications in smart homes to observe and track toddlers or oldsters for their safety,monitor indoor and outdoor activities,develop Tele immersion systems,or detect abnormal activity recognition.Three dimensions(3D)skeleton data is robust and somehow view-invariant.Due to this,it is one of the popular choices for human action recognition.This paper proposed using a transversal tree from 3D skeleton data to represent videos in a sequence.Further proposed two neural networks:convolutional neural network recurrent neural network_1(CNN_RNN_1),used to find the optimal features and convolutional neural network recurrent neural network network_2(CNN_RNN_2),used to classify actions.The deep neural network-based model proposed CNN_RNN_1 and CNN_RNN_2 that uses a convolutional neural network(CNN),Long short-term memory(LSTM)and Bidirectional Long shortterm memory(BiLSTM)layered.The systemefficiently achieves the desired accuracy over state-of-the-art models,i.e.,88.89%.The performance of the proposed model compared with the existing state-of-the-art models.The NTURGB+D dataset uses for analyzing experimental results.It is one of the large benchmark datasets for human activity recognition.Moreover,the comparison results show that the proposed model outperformed the state-ofthe-art models.展开更多
This paper addresses the problem of predicting population density leveraging cellular station data.As wireless communication devices are commonly used,cellular station data has become integral for estimating populatio...This paper addresses the problem of predicting population density leveraging cellular station data.As wireless communication devices are commonly used,cellular station data has become integral for estimating population figures and studying their movement,thereby implying significant contributions to urban planning.However,existing research grapples with issues pertinent to preprocessing base station data and the modeling of population prediction.To address this,we propose methodologies for preprocessing cellular station data to eliminate any irregular or redundant data.The preprocessing reveals a distinct cyclical characteristic and high-frequency variation in population shift.Further,we devise a multi-view enhancement model grounded on the Transformer(MVformer),targeting the improvement of the accuracy of extended time-series population predictions.Comparative experiments,conducted on the above-mentioned population dataset using four alternate Transformer-based models,indicate that our proposedMVformer model enhances prediction accuracy by approximately 30%for both univariate and multivariate time-series prediction assignments.The performance of this model in tasks pertaining to population prediction exhibits commendable results.展开更多
Recently, the reference functions for the synthesis and analysis of the autostereoscopic multiview and integral images in three-dimensional displays were introduced. In the current paper, we propose the wavelets to an...Recently, the reference functions for the synthesis and analysis of the autostereoscopic multiview and integral images in three-dimensional displays were introduced. In the current paper, we propose the wavelets to analyze such images. The wavelets are built on these reference functions as on the scaling functions of the wavelet analysis. The continuous wavelet transform was successfully applied to the testing wireframe binary objects. The restored locations correspond to the structure of the testing wireframe binary objects.展开更多
The use of hand gestures can be the most intuitive human-machine interaction medium.The early approaches for hand gesture recognition used device-based methods.These methods use mechanical or optical sensors attached ...The use of hand gestures can be the most intuitive human-machine interaction medium.The early approaches for hand gesture recognition used device-based methods.These methods use mechanical or optical sensors attached to a glove or markers,which hinder the natural human-machine communication.On the other hand,vision-based methods are less restrictive and allow for a more spontaneous communication without the need of an intermediary between human and machine.Therefore,vision gesture recognition has been a popular area of research for the past thirty years.Hand gesture recognition finds its application in many areas,particularly the automotive industry where advanced automotive human-machine interface(HMI)designers are using gesture recognition to improve driver and vehicle safety.However,technology advances go beyond active/passive safety and into convenience and comfort.In this context,one of America’s big three automakers has partnered with the Centre of Pattern Analysis and Machine Intelligence(CPAMI)at the University of Waterloo to investigate expanding their product segment through machine learning to provide an increased driver convenience and comfort with the particular application of hand gesture recognition for autonomous car parking.The present paper leverages the state-of-the-art deep learning and optimization techniques to develop a vision-based multiview dynamic hand gesture recognizer for a self-parking system.We propose a 3D-CNN gesture model architecture that we train on a publicly available hand gesture database.We apply transfer learning methods to fine-tune the pre-trained gesture model on custom-made data,which significantly improves the proposed system performance in a real world environment.We adapt the architecture of end-to-end solution to expand the state-of-the-art video classifier from a single image as input(fed by monocular camera)to a Multiview 360 feed,offered by a six cameras module.Finally,we optimize the proposed solution to work on a limited resource embedded platform(Nvidia Jetson TX2)that is used by automakers for vehicle-based features,without sacrificing the accuracy robustness and real time functionality of the system.展开更多
To learn the subgroup structure generated by multidimensional interaction, we propose a novel multiview subgroup integration technique based on tensor decomposition. Compared to the traditional subgroup analysis that ...To learn the subgroup structure generated by multidimensional interaction, we propose a novel multiview subgroup integration technique based on tensor decomposition. Compared to the traditional subgroup analysis that can only handle single-view heterogeneity, our proposed method achieves a greater level of homogeneity within the subgroups, leading to enhanced interpretability and predictive power. For computational readiness of the proposed method, we build an algorithm that incorporates pairwise shrinkage-encouraging penalties and ADMM techniques. Theoretically, we establish the asymptotic consistency and normality of the proposed estimators. Extensive simulation studies and real data analysis demonstrate that our proposal outperforms other methods in terms of prediction accuracy and grouping consistency. In addition, the analysis based on the proposed method indicates that intergenerational care significantly increases the risk of chronic diseases associated with diet and fatigue in all provinces while only reducing the risk of emotion-related chronic diseases in the eastern coastal and central regions of China.展开更多
Multiview video can provide more immersive perception than traditional single 2-D video. It enables both interactive free navigation applications as well as high-end autostereoscopic displays on which multiple users c...Multiview video can provide more immersive perception than traditional single 2-D video. It enables both interactive free navigation applications as well as high-end autostereoscopic displays on which multiple users can perceive genuine 3-D content without glasses. The multiview format also comprises much more visual information than classical 2-D or stereo 3-D content, which makes it possible to perform various interesting editing operations both on pixel-level and object-level. This survey provides a comprehensive review of existing multiview video synthesis and editing algorithms and applications. For each topic, the related technologies in classical 2-D image and video processing are reviewed. We then continue to the discussion of recent advanced techniques for multiview video virtual view synthesis and various interactive editing applications. Due to the ongoing progress on multiview video synthesis and editing, we can foresee more and more immersive 3-D video applications will appear in the future.展开更多
The aspect-based sentiment analysis(ABSA) consists of two subtasks—aspect term extraction and aspect sentiment prediction. Existing methods deal with both subtasks one by one in a pipeline manner, in which there lies...The aspect-based sentiment analysis(ABSA) consists of two subtasks—aspect term extraction and aspect sentiment prediction. Existing methods deal with both subtasks one by one in a pipeline manner, in which there lies some problems in performance and real application. This study investigates the end-to-end ABSA and proposes a novel multitask multiview network(MTMVN) architecture. Specifically, the architecture takes the unified ABSA as the main task with the two subtasks as auxiliary tasks. Meanwhile, the representation obtained from the branch network of the main task is regarded as the global view, whereas the representations of the two subtasks are considered two local views with different emphases. Through multitask learning, the main task can be facilitated by additional accurate aspect boundary information and sentiment polarity information. By enhancing the correlations between the views under the idea of multiview learning, the representation of the global view can be optimized to improve the overall performance of the model. The experimental results on three benchmark datasets show that the proposed method exceeds the existing pipeline methods and end-to-end methods, proving the superiority of our MTMVN architecture.展开更多
As an essential field of multimedia and computer vision,3D shape recognition has attracted much research attention in recent years.Multiview-based approaches have demonstrated their superiority in generating effective...As an essential field of multimedia and computer vision,3D shape recognition has attracted much research attention in recent years.Multiview-based approaches have demonstrated their superiority in generating effective 3D shape representations.Typical methods usually extract the multiview global features and aggregate them together to generate 3D shape descriptors.However,there exist two disadvantages:First,the mainstream methods ignore the comprehensive exploration of local information in each view.Second,many approaches roughly aggregate multiview features by adding or concatenating them together.The information loss for some discriminative characteristics limits the representation effectiveness.To address these problems,a novel architecture named region-based joint attention network(RJAN)was proposed.Specifically,the authors first design a hierarchical local information exploration module for view descriptor extraction.The region-to-region and channel-to-channel relationships from different granularities can be comprehensively explored and utilised to provide more discriminative characteristics for view feature learning.Subsequently,a novel relation-aware view aggregation module is designed to aggregate the multiview features for shape descriptor generation,considering the view-to-view relationships.Extensive experiments were conducted on three public databases:ModelNet40,ModelNet10,and ShapeNetCore55.RJAN achieves state-of-the-art performance in the tasks of 3D shape classification and 3D shape retrieval,which demonstrates the effectiveness of RJAN.The code has been released on https://github.com/slurrpp/RJAN.展开更多
基金Supported by the European Union’s Horizon 2020 Research and Innovation Programme under Marie Skłodowska-Curie grant agreement(No.956770)the Academy of Finland under Grant 325530.
文摘Stereoscopic and multiview rendering are used for virtual reality and the synthetic generation of light fields from three-dimensional scenes.Because rendering multiple views using ray tracing techniques is computationally expensive,the utilization of multiprocessor machines is necessary to achieve real-time frame rates.In this study,we propose a dynamic load-balancing algorithm for real-time multiview path tracing on multi-compute device platforms.The proposed algorithm was adapted to heterogeneous hardware combinations and dynamic scenes in real time.We show that on a heterogeneous dual-GPU platform,our implementation reduces the rendering time by an average of approximately 30%–50%compared with that of a uniform workload distribution,depending on the scene and number of views.
基金The authors extend their appreciation to the Deanship of Scientific Research at Imam Mohammad Ibn Saud Islamic University for funding this work through Research Group no.RG-21-07-01.
文摘Traffic accidents are caused by driver fatigue or distraction in many cases.To prevent accidents,several low-cost hypovigilance(hypo-V)systems were developed in the past based on a multimodal-hybrid(physiological and behavioral)feature set.Similarly in this paper,real-time driver inattention and fatigue(Hypo-Driver)detection system is proposed through multi-view cameras and biosignal sensors to extract hybrid features.The considered features are derived from non-intrusive sensors that are related to the changes in driving behavior and visual facial expressions.To get enhanced visual facial features in uncontrolled environment,three cameras are deployed on multiview points(0◦,45◦,and 90◦)of the drivers.To develop a Hypo-Driver system,the physiological signals(electroencephalography(EEG),electrocardiography(ECG),electro-myography(sEMG),and electrooculography(EOG))and behavioral information(PERCLOS70-80-90%,mouth aspect ratio(MAR),eye aspect ratio(EAR),blinking frequency(BF),head-titled ratio(HT-R))are collected and pre-processed,then followed by feature selection and fusion techniques.The driver behaviors are classified into five stages such as normal,fatigue,visual inattention,cognitive inattention,and drowsy.This improved hypo-Driver system utilized trained behavioral features by a convolutional neural network(CNNs),recurrent neural network and long short-term memory(RNN-LSTM)model is used to extract physiological features.After fusion of these features,the Hypo-Driver system is classified hypo-V into five stages based on trained layers and dropout-layer in the deep-residual neural network(DRNN)model.To test the performance of a hypo-Driver system,data from 20 drivers are acquired.The results of Hypo-Driver compared to state-of-theart methods are presented.Compared to the state-of-the-art Hypo-V system,on average,the Hypo-Driver system achieved a detection accuracy(AC)of 96.5%.The obtained results indicate that the Hypo-Driver system based on multimodal and multiview features outperforms other state-of-the-art driver Hypo-V systems by handling many anomalies.
文摘Human activity recognition is a recent area of research for researchers.Activity recognition has many applications in smart homes to observe and track toddlers or oldsters for their safety,monitor indoor and outdoor activities,develop Tele immersion systems,or detect abnormal activity recognition.Three dimensions(3D)skeleton data is robust and somehow view-invariant.Due to this,it is one of the popular choices for human action recognition.This paper proposed using a transversal tree from 3D skeleton data to represent videos in a sequence.Further proposed two neural networks:convolutional neural network recurrent neural network_1(CNN_RNN_1),used to find the optimal features and convolutional neural network recurrent neural network network_2(CNN_RNN_2),used to classify actions.The deep neural network-based model proposed CNN_RNN_1 and CNN_RNN_2 that uses a convolutional neural network(CNN),Long short-term memory(LSTM)and Bidirectional Long shortterm memory(BiLSTM)layered.The systemefficiently achieves the desired accuracy over state-of-the-art models,i.e.,88.89%.The performance of the proposed model compared with the existing state-of-the-art models.The NTURGB+D dataset uses for analyzing experimental results.It is one of the large benchmark datasets for human activity recognition.Moreover,the comparison results show that the proposed model outperformed the state-ofthe-art models.
基金Guangdong Basic and Applied Basic Research Foundation under Grant No.2024A1515012485in part by the Shenzhen Fundamental Research Program under Grant JCYJ20220810112354002.
文摘This paper addresses the problem of predicting population density leveraging cellular station data.As wireless communication devices are commonly used,cellular station data has become integral for estimating population figures and studying their movement,thereby implying significant contributions to urban planning.However,existing research grapples with issues pertinent to preprocessing base station data and the modeling of population prediction.To address this,we propose methodologies for preprocessing cellular station data to eliminate any irregular or redundant data.The preprocessing reveals a distinct cyclical characteristic and high-frequency variation in population shift.Further,we devise a multi-view enhancement model grounded on the Transformer(MVformer),targeting the improvement of the accuracy of extended time-series population predictions.Comparative experiments,conducted on the above-mentioned population dataset using four alternate Transformer-based models,indicate that our proposedMVformer model enhances prediction accuracy by approximately 30%for both univariate and multivariate time-series prediction assignments.The performance of this model in tasks pertaining to population prediction exhibits commendable results.
文摘Recently, the reference functions for the synthesis and analysis of the autostereoscopic multiview and integral images in three-dimensional displays were introduced. In the current paper, we propose the wavelets to analyze such images. The wavelets are built on these reference functions as on the scaling functions of the wavelet analysis. The continuous wavelet transform was successfully applied to the testing wireframe binary objects. The restored locations correspond to the structure of the testing wireframe binary objects.
文摘The use of hand gestures can be the most intuitive human-machine interaction medium.The early approaches for hand gesture recognition used device-based methods.These methods use mechanical or optical sensors attached to a glove or markers,which hinder the natural human-machine communication.On the other hand,vision-based methods are less restrictive and allow for a more spontaneous communication without the need of an intermediary between human and machine.Therefore,vision gesture recognition has been a popular area of research for the past thirty years.Hand gesture recognition finds its application in many areas,particularly the automotive industry where advanced automotive human-machine interface(HMI)designers are using gesture recognition to improve driver and vehicle safety.However,technology advances go beyond active/passive safety and into convenience and comfort.In this context,one of America’s big three automakers has partnered with the Centre of Pattern Analysis and Machine Intelligence(CPAMI)at the University of Waterloo to investigate expanding their product segment through machine learning to provide an increased driver convenience and comfort with the particular application of hand gesture recognition for autonomous car parking.The present paper leverages the state-of-the-art deep learning and optimization techniques to develop a vision-based multiview dynamic hand gesture recognizer for a self-parking system.We propose a 3D-CNN gesture model architecture that we train on a publicly available hand gesture database.We apply transfer learning methods to fine-tune the pre-trained gesture model on custom-made data,which significantly improves the proposed system performance in a real world environment.We adapt the architecture of end-to-end solution to expand the state-of-the-art video classifier from a single image as input(fed by monocular camera)to a Multiview 360 feed,offered by a six cameras module.Finally,we optimize the proposed solution to work on a limited resource embedded platform(Nvidia Jetson TX2)that is used by automakers for vehicle-based features,without sacrificing the accuracy robustness and real time functionality of the system.
基金supported by National Key R&D Program of China (Grant No. 2022YFA1003702)National Natural Science Foundation of China (Grant Nos. 11931014 and 12271441)New Cornerstone Science Foundation
文摘To learn the subgroup structure generated by multidimensional interaction, we propose a novel multiview subgroup integration technique based on tensor decomposition. Compared to the traditional subgroup analysis that can only handle single-view heterogeneity, our proposed method achieves a greater level of homogeneity within the subgroups, leading to enhanced interpretability and predictive power. For computational readiness of the proposed method, we build an algorithm that incorporates pairwise shrinkage-encouraging penalties and ADMM techniques. Theoretically, we establish the asymptotic consistency and normality of the proposed estimators. Extensive simulation studies and real data analysis demonstrate that our proposal outperforms other methods in terms of prediction accuracy and grouping consistency. In addition, the analysis based on the proposed method indicates that intergenerational care significantly increases the risk of chronic diseases associated with diet and fatigue in all provinces while only reducing the risk of emotion-related chronic diseases in the eastern coastal and central regions of China.
基金partially supported by Innoviris(3-DLicornea project)FWO(project G.0256.15)+3 种基金supported by the National Natural Science Foundation of China(Nos.61272226 and 61373069)Research Grant of Beijing Higher Institution Engineering Research CenterTsinghua-Tencent Joint Laboratory for Internet Innovation TechnologyTsinghua University Initiative Scientific Research Program
文摘Multiview video can provide more immersive perception than traditional single 2-D video. It enables both interactive free navigation applications as well as high-end autostereoscopic displays on which multiple users can perceive genuine 3-D content without glasses. The multiview format also comprises much more visual information than classical 2-D or stereo 3-D content, which makes it possible to perform various interesting editing operations both on pixel-level and object-level. This survey provides a comprehensive review of existing multiview video synthesis and editing algorithms and applications. For each topic, the related technologies in classical 2-D image and video processing are reviewed. We then continue to the discussion of recent advanced techniques for multiview video virtual view synthesis and various interactive editing applications. Due to the ongoing progress on multiview video synthesis and editing, we can foresee more and more immersive 3-D video applications will appear in the future.
基金supported by the National Natural Science Foundation of China(No.61976247)
文摘The aspect-based sentiment analysis(ABSA) consists of two subtasks—aspect term extraction and aspect sentiment prediction. Existing methods deal with both subtasks one by one in a pipeline manner, in which there lies some problems in performance and real application. This study investigates the end-to-end ABSA and proposes a novel multitask multiview network(MTMVN) architecture. Specifically, the architecture takes the unified ABSA as the main task with the two subtasks as auxiliary tasks. Meanwhile, the representation obtained from the branch network of the main task is regarded as the global view, whereas the representations of the two subtasks are considered two local views with different emphases. Through multitask learning, the main task can be facilitated by additional accurate aspect boundary information and sentiment polarity information. By enhancing the correlations between the views under the idea of multiview learning, the representation of the global view can be optimized to improve the overall performance of the model. The experimental results on three benchmark datasets show that the proposed method exceeds the existing pipeline methods and end-to-end methods, proving the superiority of our MTMVN architecture.
基金the National Key Research and Development Program of China,Grant/Award Number:2020YFB1711704the National Natural Science Foundation of China,Grant/Award Number:62272337。
文摘As an essential field of multimedia and computer vision,3D shape recognition has attracted much research attention in recent years.Multiview-based approaches have demonstrated their superiority in generating effective 3D shape representations.Typical methods usually extract the multiview global features and aggregate them together to generate 3D shape descriptors.However,there exist two disadvantages:First,the mainstream methods ignore the comprehensive exploration of local information in each view.Second,many approaches roughly aggregate multiview features by adding or concatenating them together.The information loss for some discriminative characteristics limits the representation effectiveness.To address these problems,a novel architecture named region-based joint attention network(RJAN)was proposed.Specifically,the authors first design a hierarchical local information exploration module for view descriptor extraction.The region-to-region and channel-to-channel relationships from different granularities can be comprehensively explored and utilised to provide more discriminative characteristics for view feature learning.Subsequently,a novel relation-aware view aggregation module is designed to aggregate the multiview features for shape descriptor generation,considering the view-to-view relationships.Extensive experiments were conducted on three public databases:ModelNet40,ModelNet10,and ShapeNetCore55.RJAN achieves state-of-the-art performance in the tasks of 3D shape classification and 3D shape retrieval,which demonstrates the effectiveness of RJAN.The code has been released on https://github.com/slurrpp/RJAN.
