Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of ...Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity.Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process.However,it takes a lot of time and effort for researchers to use the existing activation function in their research.Therefore,in this paper,we propose a universal activation function(UA)so that researchers can easily create and apply various activation functions and improve the performance of neural networks.UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters.The famous Convolutional Neural Network(CNN)and benchmark datasetwere used to evaluate the experimental performance of the UA proposed in this study.We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which theUA is applied.In addition,we evaluated the performance of the new activation function generated by adjusting the hyperparameters of theUA.The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5%through the UA,although most of them showed similar performance to the traditional activation function.展开更多
Due to the lack of large-scale emotion databases,it is hard to obtain comparable improvement in multimodal emotion recognition of the deep neural network by deep learning,which has made great progress in other areas.W...Due to the lack of large-scale emotion databases,it is hard to obtain comparable improvement in multimodal emotion recognition of the deep neural network by deep learning,which has made great progress in other areas.We use transfer learning to improve its performance with pretrained models on largescale data.Audio is encoded using deep speech recognition networks with 500 hours’speech and video is encoded using convolutional neural networks with over 110,000 images.The extracted audio and visual features are fed into Long Short-Term Memory to train models respectively.Logistic regression and ensemble method are performed in decision level fusion.The experiment results indicate that 1)audio features extracted from deep speech recognition networks achieve better performance than handcrafted audio features;2)the visual emotion recognition obtains better performance than audio emotion recognition;3)the ensemble method gets better performance than logistic regression and prior knowledge from micro-F1 value further improves the performance and robustness,achieving accuracy of 67.00%for“happy”,54.90%for“an?gry”,and 51.69%for“sad”.展开更多
Smart Grids(SG)is a power system development concept that has received significant attention nationally.SG signifies real-time data for specific communication requirements.The best capabilities for monitoring and control...Smart Grids(SG)is a power system development concept that has received significant attention nationally.SG signifies real-time data for specific communication requirements.The best capabilities for monitoring and controlling the grid are essential to system stability.One of the most critical needs for smart-grid execution is fast,precise,and economically synchronized measurements,which are made feasible by Phasor Measurement Units(PMU).PMUs can pro-vide synchronized measurements and measure voltages as well as current phasors dynamically.PMUs utilize GPS time-stamping at Coordinated Universal Time(UTC)to capture electric phasors with great accuracy and precision.This research tends to Deep Learning(DL)advances to design a Residual Network(ResNet)model that can accurately identify and classify defects in grid-connected systems.As part of fault detection and probe,the proposed strategy uses a ResNet-50 tech-nique to evaluate real-time measurement data from geographically scattered PMUs.As a result of its excellent signal classification efficiency and ability to extract high-quality signal features,its fault diagnosis performance is excellent.Our results demonstrate that the proposed method is effective in detecting and classifying faults at sufficient time.The proposed approaches classify the fault type with a precision of 98.5%and an accuracy of 99.1%.The long-short-term memory(LSTM),Convolutional Neural Network(CNN),and CNN-LSTM algo-rithms are applied to compare the networks.Real-world data tends to evaluate these networks.展开更多
Recursively embedded atom neural network(REANN)is a general-purpose atomistic machine learning software package for representing potential energy and other physical properties.The original REANN 1.0 architecture is a ...Recursively embedded atom neural network(REANN)is a general-purpose atomistic machine learning software package for representing potential energy and other physical properties.The original REANN 1.0 architecture is a physically inspired invariant message passing neural network,which was designed for systems with a limited number of elements.It is efficient but hardly transferable to more complex multi-element systems.In this work,we release REANN 2.0 aimed at multi-element systems and universal potentials,which integrates element embedding and equivariant representation.Compared to the first version,REANN 2.0 demonstrates enhanced ele-ment transferability and higher accuracy across various periodic systems with higher efficiency.Built upon this framework,a pre-trained REANN-MPtrj model without fine-tuning accurately predicts the lithium-ion diffusion dynamics in a benchmark solid-state electrolyte Li_(3)YCl_(6).We hope this open-source software package will facilitate the development of computationally efficient universal potentials in the future.展开更多
As various types of data grow explosively,largescale data storage,backup,and transmission become challenging,which motivates many researchers to propose efficient universal compression algorithms for multi-source data...As various types of data grow explosively,largescale data storage,backup,and transmission become challenging,which motivates many researchers to propose efficient universal compression algorithms for multi-source data.In recent years,due to the emergence of hardware acceleration devices such as GPUs,TPUs,DPUs,and FPGAs,the performance bottleneck of neural networks(NN)has been overcome,making NN-based compression algorithms increasingly practical and popular.However,the research survey for the NN-based universal lossless compressors has not been conducted yet,and there is also a lack of unified evaluation metrics.To address the above problems,in this paper,we present a holistic survey as well as benchmark evaluations.Specifically,i)we thoroughly investigate NNbased lossless universal compression algorithms toward multisource data and classify them into 3 types:static pre-training,adaptive,and semi-adaptive.ii)We unify 19 evaluation metrics to comprehensively assess the compression effect,resource consumption,and model performance of compressors.iii)We conduct experiments more than 4600 CPU/GPU hours to evaluate 17 state-of-the-art compressors on 28 real-world datasets across data types of text,images,videos,audio,etc.iv)We also summarize the strengths and drawbacks of NNbased lossless data compressors and discuss promising research directions.We summarize the results as the NN-based Lossless Compressors Benchmark(NNLCB,See fahaihi.github.io/NNLCB website),which will be updated and maintained continuously in the future.展开更多
Light-field imaging has wide applications in various domains,including microscale life science imaging,mesoscale neuroimaging,and macroscale fluid dynamics imaging.The development of deep learning-based reconstruction...Light-field imaging has wide applications in various domains,including microscale life science imaging,mesoscale neuroimaging,and macroscale fluid dynamics imaging.The development of deep learning-based reconstruction methods has greatly facilitated high-resolution light-field image processing,however,current deep learning-based light-field reconstruction methods have predominantly concentrated on the microscale.Considering the multiscale imaging capacity of light-field technique,a network that can work over variant scales of light-field image reconstruction will significantly benefit the development of volumetric imaging.Unfortunately,to our knowledge,no one has reported a universal high-resolution light-field image reconstruction algorithm that is compatible with microscale,mesoscale,and macroscale.To fill this gap,we present a real-time and universal network(RTU-Net)to reconstruct high-resolution light-field images at any scale.RTU-Net,as the first network that works over multiscale light-field image reconstruction,employs an adaptive loss function based on generative adversarial theory and consequently exhibits strong generalization capability.We comprehensively assessed the performance of RTU-Net through the reconstruction of multiscale light-field images,including microscale tubulin and mitochondrion dataset,mesoscale synthetic mouse neuro dataset,and macroscale light-field particle imaging velocimetry dataset.The results indicated that RTU-Net has achieved real-time and high-resolution light-field image reconstruction for volume sizes ranging from 300μm×300μm×12μm to 25 mm×25 mm×25 mm,and demonstrated higher resolution when compared with recently reported light-field reconstruction networks.The high-resolution,strong robustness,high efficiency,and especially the general applicability of RTU-Net will significantly deepen our insight into high-resolution and volumetric imaging.展开更多
We present a full space inverse materials design(FSIMD)approach that fully automates the materials design for target physical properties without the need to provide the atomic composition,chemical stoichiometry,and cr...We present a full space inverse materials design(FSIMD)approach that fully automates the materials design for target physical properties without the need to provide the atomic composition,chemical stoichiometry,and crystal structure in advance.Here,we used density functional theory reference data to train a universal machine learning potential(UPot)and transfer learning to train a universal bulk modulus model(UBmod).Both UPot and UBmod were able to cover materials systems composed of any element among 42 elements.Interfaced with optimization algorithm and enhanced sampling,the FSIMD approach is applied to find the materials with the largest cohesive energy and the largest bulk modulus,respectively.NaCl-type ZrC was found to be the material with the largest cohesive energy.For bulk modulus,diamond was identified to have the largest value.The FSIMD approach is also applied to design materials with other multi-objective properties with accuracy limited principally by the amount,reliability,and diversity of the training data.The FSIMD approach provides a new way for inverse materials design with other functional properties for practical applications.展开更多
Relic gravitational waves(RGWs)from the early Universe carry crucial and fundamental cosmological information.Therefore,it is of extraordinary importance to investigate potential RGW signals in the data from observato...Relic gravitational waves(RGWs)from the early Universe carry crucial and fundamental cosmological information.Therefore,it is of extraordinary importance to investigate potential RGW signals in the data from observatories such as the LIGO-Virgo-KAGRA network.Here,focusing on typical RGWs from the inflation and the first-order phase transition(by sound waves and bubble collisions),effective and targeted deep learning neural networks are established to search for these RGW signals within the real LIGO data(O2,O3a and O3b).Through adjustment and adaptation processes,we develop suitable Convolutional Neural Networks(CNNs)to estimate the likelihood(characterized by quantitative values and distributions)that the focused RGW signals are present in the LIGO data.We find that if the constructed CNN properly estimates the parameters of the RGWs,it can determine with high accuracy(approximately 94%to 99%)whether the samples contain such RGW signals;otherwise,the likelihood provided by the CNN cannot be considered reliable.After testing a large amount of LIGO data,the findings show no evidence of RGWs from:1)inflation,2)sound waves,or 3)bubble collisions,as predicted by the focused theories.The results also provide upper limits of their GW spectral energy densities of h^(2)Ω_(gw)~10^(-5),respectively for parameter boundaries within 1)[β∈(-1.87,-1.85)×α∈(0.005,0.007)],2)[β/H_(pt)∈(0.02,0.16)×α∈(1,10)×T_(pt)∈(5*10^(9),10^(10))Gev],and 3)[β/H_(pt)∈(0.08,0.2)×α∈(1,10)×T_(pt)∈(5*10^(9),8*10^(10))Gev].In short,null results and upper limits are obtained,and the analysis suggests that our developed methods and neural networks to search for typical RGWs in the LIGO data are effective and reliable,providing a viable scheme for exploring possible RGWs from the early Universe and placing constraints on relevant cosmological theories.展开更多
基金This work was supported by the National Research Foundation of Korea(NRF)grant funded by the Korea government(MSIT)(No.2022R1F1A1062953).
文摘Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity.Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process.However,it takes a lot of time and effort for researchers to use the existing activation function in their research.Therefore,in this paper,we propose a universal activation function(UA)so that researchers can easily create and apply various activation functions and improve the performance of neural networks.UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters.The famous Convolutional Neural Network(CNN)and benchmark datasetwere used to evaluate the experimental performance of the UA proposed in this study.We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which theUA is applied.In addition,we evaluated the performance of the new activation function generated by adjusting the hyperparameters of theUA.The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5%through the UA,although most of them showed similar performance to the traditional activation function.
文摘Due to the lack of large-scale emotion databases,it is hard to obtain comparable improvement in multimodal emotion recognition of the deep neural network by deep learning,which has made great progress in other areas.We use transfer learning to improve its performance with pretrained models on largescale data.Audio is encoded using deep speech recognition networks with 500 hours’speech and video is encoded using convolutional neural networks with over 110,000 images.The extracted audio and visual features are fed into Long Short-Term Memory to train models respectively.Logistic regression and ensemble method are performed in decision level fusion.The experiment results indicate that 1)audio features extracted from deep speech recognition networks achieve better performance than handcrafted audio features;2)the visual emotion recognition obtains better performance than audio emotion recognition;3)the ensemble method gets better performance than logistic regression and prior knowledge from micro-F1 value further improves the performance and robustness,achieving accuracy of 67.00%for“happy”,54.90%for“an?gry”,and 51.69%for“sad”.
文摘Smart Grids(SG)is a power system development concept that has received significant attention nationally.SG signifies real-time data for specific communication requirements.The best capabilities for monitoring and controlling the grid are essential to system stability.One of the most critical needs for smart-grid execution is fast,precise,and economically synchronized measurements,which are made feasible by Phasor Measurement Units(PMU).PMUs can pro-vide synchronized measurements and measure voltages as well as current phasors dynamically.PMUs utilize GPS time-stamping at Coordinated Universal Time(UTC)to capture electric phasors with great accuracy and precision.This research tends to Deep Learning(DL)advances to design a Residual Network(ResNet)model that can accurately identify and classify defects in grid-connected systems.As part of fault detection and probe,the proposed strategy uses a ResNet-50 tech-nique to evaluate real-time measurement data from geographically scattered PMUs.As a result of its excellent signal classification efficiency and ability to extract high-quality signal features,its fault diagnosis performance is excellent.Our results demonstrate that the proposed method is effective in detecting and classifying faults at sufficient time.The proposed approaches classify the fault type with a precision of 98.5%and an accuracy of 99.1%.The long-short-term memory(LSTM),Convolutional Neural Network(CNN),and CNN-LSTM algo-rithms are applied to compare the networks.Real-world data tends to evaluate these networks.
基金the support by the Strategic Priority Research Program of the Chinese Academy of Sciences(XDB0450101)the National Natural Science Foundation of China(Nos.22325304,22221003 and 22033007)。
文摘Recursively embedded atom neural network(REANN)is a general-purpose atomistic machine learning software package for representing potential energy and other physical properties.The original REANN 1.0 architecture is a physically inspired invariant message passing neural network,which was designed for systems with a limited number of elements.It is efficient but hardly transferable to more complex multi-element systems.In this work,we release REANN 2.0 aimed at multi-element systems and universal potentials,which integrates element embedding and equivariant representation.Compared to the first version,REANN 2.0 demonstrates enhanced ele-ment transferability and higher accuracy across various periodic systems with higher efficiency.Built upon this framework,a pre-trained REANN-MPtrj model without fine-tuning accurately predicts the lithium-ion diffusion dynamics in a benchmark solid-state electrolyte Li_(3)YCl_(6).We hope this open-source software package will facilitate the development of computationally efficient universal potentials in the future.
基金supported by the National Natural Science Foundation of China(Grant Nos.62272253 and 62272252)the Fundamental Research Funds for the Central Universities.It was also supported in part by the China Scholarship Council(CSC202406200085)the Innovation Project of Guangxi Graduate Education(YCBZ2024005).
文摘As various types of data grow explosively,largescale data storage,backup,and transmission become challenging,which motivates many researchers to propose efficient universal compression algorithms for multi-source data.In recent years,due to the emergence of hardware acceleration devices such as GPUs,TPUs,DPUs,and FPGAs,the performance bottleneck of neural networks(NN)has been overcome,making NN-based compression algorithms increasingly practical and popular.However,the research survey for the NN-based universal lossless compressors has not been conducted yet,and there is also a lack of unified evaluation metrics.To address the above problems,in this paper,we present a holistic survey as well as benchmark evaluations.Specifically,i)we thoroughly investigate NNbased lossless universal compression algorithms toward multisource data and classify them into 3 types:static pre-training,adaptive,and semi-adaptive.ii)We unify 19 evaluation metrics to comprehensively assess the compression effect,resource consumption,and model performance of compressors.iii)We conduct experiments more than 4600 CPU/GPU hours to evaluate 17 state-of-the-art compressors on 28 real-world datasets across data types of text,images,videos,audio,etc.iv)We also summarize the strengths and drawbacks of NNbased lossless data compressors and discuss promising research directions.We summarize the results as the NN-based Lossless Compressors Benchmark(NNLCB,See fahaihi.github.io/NNLCB website),which will be updated and maintained continuously in the future.
基金supported by National Natural Science Foundation of China(12402336,82201637,U20A2070,and 12025202)National High-Level Talent Project(YQR23069)+6 种基金Natural Science Foundation of Jiangsu Province(BK20230876)the Young Elite Scientist Sponsorship Program by CAST(YESS20210238)Forwardlooking layout projects(1002-ILB24009)Zhejang Provincial Medical and Health Technology Project(Grant No.2024KY246,2025KY180)Scientific Research Foundation of Hangzhou City University(No.J-202402)Open Research Fund of the State Key Laboratory of Brain-Machine Intelligence,Zhejiang University(Grant No.BMI2400025)Hangzhou Science and Technology Bureau.
文摘Light-field imaging has wide applications in various domains,including microscale life science imaging,mesoscale neuroimaging,and macroscale fluid dynamics imaging.The development of deep learning-based reconstruction methods has greatly facilitated high-resolution light-field image processing,however,current deep learning-based light-field reconstruction methods have predominantly concentrated on the microscale.Considering the multiscale imaging capacity of light-field technique,a network that can work over variant scales of light-field image reconstruction will significantly benefit the development of volumetric imaging.Unfortunately,to our knowledge,no one has reported a universal high-resolution light-field image reconstruction algorithm that is compatible with microscale,mesoscale,and macroscale.To fill this gap,we present a real-time and universal network(RTU-Net)to reconstruct high-resolution light-field images at any scale.RTU-Net,as the first network that works over multiscale light-field image reconstruction,employs an adaptive loss function based on generative adversarial theory and consequently exhibits strong generalization capability.We comprehensively assessed the performance of RTU-Net through the reconstruction of multiscale light-field images,including microscale tubulin and mitochondrion dataset,mesoscale synthetic mouse neuro dataset,and macroscale light-field particle imaging velocimetry dataset.The results indicated that RTU-Net has achieved real-time and high-resolution light-field image reconstruction for volume sizes ranging from 300μm×300μm×12μm to 25 mm×25 mm×25 mm,and demonstrated higher resolution when compared with recently reported light-field reconstruction networks.The high-resolution,strong robustness,high efficiency,and especially the general applicability of RTU-Net will significantly deepen our insight into high-resolution and volumetric imaging.
基金funding support by the National Key Research and Development Program of China(2020YFB1506400)the National Natural Science Foundation of China(11974257 and 12188101)+1 种基金Jiangsu Distinguished Young Talent Funding(BK20200003)Soochow Municipal Laboratory for low carbon technologies and industries.
文摘We present a full space inverse materials design(FSIMD)approach that fully automates the materials design for target physical properties without the need to provide the atomic composition,chemical stoichiometry,and crystal structure in advance.Here,we used density functional theory reference data to train a universal machine learning potential(UPot)and transfer learning to train a universal bulk modulus model(UBmod).Both UPot and UBmod were able to cover materials systems composed of any element among 42 elements.Interfaced with optimization algorithm and enhanced sampling,the FSIMD approach is applied to find the materials with the largest cohesive energy and the largest bulk modulus,respectively.NaCl-type ZrC was found to be the material with the largest cohesive energy.For bulk modulus,diamond was identified to have the largest value.The FSIMD approach is also applied to design materials with other multi-objective properties with accuracy limited principally by the amount,reliability,and diversity of the training data.The FSIMD approach provides a new way for inverse materials design with other functional properties for practical applications.
基金supported in part by the National Natural Science Foundation of China under Grant Nos.11605015,12347101 and 12147102the Natural Scienceof Chongqing under Grant No.cstc2020jcyjmsxm X0944the Research Funds for the Central Universities under Grant No.2022CDJXY-002。
文摘Relic gravitational waves(RGWs)from the early Universe carry crucial and fundamental cosmological information.Therefore,it is of extraordinary importance to investigate potential RGW signals in the data from observatories such as the LIGO-Virgo-KAGRA network.Here,focusing on typical RGWs from the inflation and the first-order phase transition(by sound waves and bubble collisions),effective and targeted deep learning neural networks are established to search for these RGW signals within the real LIGO data(O2,O3a and O3b).Through adjustment and adaptation processes,we develop suitable Convolutional Neural Networks(CNNs)to estimate the likelihood(characterized by quantitative values and distributions)that the focused RGW signals are present in the LIGO data.We find that if the constructed CNN properly estimates the parameters of the RGWs,it can determine with high accuracy(approximately 94%to 99%)whether the samples contain such RGW signals;otherwise,the likelihood provided by the CNN cannot be considered reliable.After testing a large amount of LIGO data,the findings show no evidence of RGWs from:1)inflation,2)sound waves,or 3)bubble collisions,as predicted by the focused theories.The results also provide upper limits of their GW spectral energy densities of h^(2)Ω_(gw)~10^(-5),respectively for parameter boundaries within 1)[β∈(-1.87,-1.85)×α∈(0.005,0.007)],2)[β/H_(pt)∈(0.02,0.16)×α∈(1,10)×T_(pt)∈(5*10^(9),10^(10))Gev],and 3)[β/H_(pt)∈(0.08,0.2)×α∈(1,10)×T_(pt)∈(5*10^(9),8*10^(10))Gev].In short,null results and upper limits are obtained,and the analysis suggests that our developed methods and neural networks to search for typical RGWs in the LIGO data are effective and reliable,providing a viable scheme for exploring possible RGWs from the early Universe and placing constraints on relevant cosmological theories.
