Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of ...Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity.Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process.However,it takes a lot of time and effort for researchers to use the existing activation function in their research.Therefore,in this paper,we propose a universal activation function(UA)so that researchers can easily create and apply various activation functions and improve the performance of neural networks.UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters.The famous Convolutional Neural Network(CNN)and benchmark datasetwere used to evaluate the experimental performance of the UA proposed in this study.We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which theUA is applied.In addition,we evaluated the performance of the new activation function generated by adjusting the hyperparameters of theUA.The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5%through the UA,although most of them showed similar performance to the traditional activation function.展开更多
Due to the lack of large-scale emotion databases,it is hard to obtain comparable improvement in multimodal emotion recognition of the deep neural network by deep learning,which has made great progress in other areas.W...Due to the lack of large-scale emotion databases,it is hard to obtain comparable improvement in multimodal emotion recognition of the deep neural network by deep learning,which has made great progress in other areas.We use transfer learning to improve its performance with pretrained models on largescale data.Audio is encoded using deep speech recognition networks with 500 hours’speech and video is encoded using convolutional neural networks with over 110,000 images.The extracted audio and visual features are fed into Long Short-Term Memory to train models respectively.Logistic regression and ensemble method are performed in decision level fusion.The experiment results indicate that 1)audio features extracted from deep speech recognition networks achieve better performance than handcrafted audio features;2)the visual emotion recognition obtains better performance than audio emotion recognition;3)the ensemble method gets better performance than logistic regression and prior knowledge from micro-F1 value further improves the performance and robustness,achieving accuracy of 67.00%for“happy”,54.90%for“an?gry”,and 51.69%for“sad”.展开更多
Smart Grids(SG)is a power system development concept that has received significant attention nationally.SG signifies real-time data for specific communication requirements.The best capabilities for monitoring and control...Smart Grids(SG)is a power system development concept that has received significant attention nationally.SG signifies real-time data for specific communication requirements.The best capabilities for monitoring and controlling the grid are essential to system stability.One of the most critical needs for smart-grid execution is fast,precise,and economically synchronized measurements,which are made feasible by Phasor Measurement Units(PMU).PMUs can pro-vide synchronized measurements and measure voltages as well as current phasors dynamically.PMUs utilize GPS time-stamping at Coordinated Universal Time(UTC)to capture electric phasors with great accuracy and precision.This research tends to Deep Learning(DL)advances to design a Residual Network(ResNet)model that can accurately identify and classify defects in grid-connected systems.As part of fault detection and probe,the proposed strategy uses a ResNet-50 tech-nique to evaluate real-time measurement data from geographically scattered PMUs.As a result of its excellent signal classification efficiency and ability to extract high-quality signal features,its fault diagnosis performance is excellent.Our results demonstrate that the proposed method is effective in detecting and classifying faults at sufficient time.The proposed approaches classify the fault type with a precision of 98.5%and an accuracy of 99.1%.The long-short-term memory(LSTM),Convolutional Neural Network(CNN),and CNN-LSTM algo-rithms are applied to compare the networks.Real-world data tends to evaluate these networks.展开更多
As various types of data grow explosively,largescale data storage,backup,and transmission become challenging,which motivates many researchers to propose efficient universal compression algorithms for multi-source data...As various types of data grow explosively,largescale data storage,backup,and transmission become challenging,which motivates many researchers to propose efficient universal compression algorithms for multi-source data.In recent years,due to the emergence of hardware acceleration devices such as GPUs,TPUs,DPUs,and FPGAs,the performance bottleneck of neural networks(NN)has been overcome,making NN-based compression algorithms increasingly practical and popular.However,the research survey for the NN-based universal lossless compressors has not been conducted yet,and there is also a lack of unified evaluation metrics.To address the above problems,in this paper,we present a holistic survey as well as benchmark evaluations.Specifically,i)we thoroughly investigate NNbased lossless universal compression algorithms toward multisource data and classify them into 3 types:static pre-training,adaptive,and semi-adaptive.ii)We unify 19 evaluation metrics to comprehensively assess the compression effect,resource consumption,and model performance of compressors.iii)We conduct experiments more than 4600 CPU/GPU hours to evaluate 17 state-of-the-art compressors on 28 real-world datasets across data types of text,images,videos,audio,etc.iv)We also summarize the strengths and drawbacks of NNbased lossless data compressors and discuss promising research directions.We summarize the results as the NN-based Lossless Compressors Benchmark(NNLCB,See fahaihi.github.io/NNLCB website),which will be updated and maintained continuously in the future.展开更多
Light-field imaging has wide applications in various domains,including microscale life science imaging,mesoscale neuroimaging,and macroscale fluid dynamics imaging.The development of deep learning-based reconstruction...Light-field imaging has wide applications in various domains,including microscale life science imaging,mesoscale neuroimaging,and macroscale fluid dynamics imaging.The development of deep learning-based reconstruction methods has greatly facilitated high-resolution light-field image processing,however,current deep learning-based light-field reconstruction methods have predominantly concentrated on the microscale.Considering the multiscale imaging capacity of light-field technique,a network that can work over variant scales of light-field image reconstruction will significantly benefit the development of volumetric imaging.Unfortunately,to our knowledge,no one has reported a universal high-resolution light-field image reconstruction algorithm that is compatible with microscale,mesoscale,and macroscale.To fill this gap,we present a real-time and universal network(RTU-Net)to reconstruct high-resolution light-field images at any scale.RTU-Net,as the first network that works over multiscale light-field image reconstruction,employs an adaptive loss function based on generative adversarial theory and consequently exhibits strong generalization capability.We comprehensively assessed the performance of RTU-Net through the reconstruction of multiscale light-field images,including microscale tubulin and mitochondrion dataset,mesoscale synthetic mouse neuro dataset,and macroscale light-field particle imaging velocimetry dataset.The results indicated that RTU-Net has achieved real-time and high-resolution light-field image reconstruction for volume sizes ranging from 300μm×300μm×12μm to 25 mm×25 mm×25 mm,and demonstrated higher resolution when compared with recently reported light-field reconstruction networks.The high-resolution,strong robustness,high efficiency,and especially the general applicability of RTU-Net will significantly deepen our insight into high-resolution and volumetric imaging.展开更多
We present a full space inverse materials design(FSIMD)approach that fully automates the materials design for target physical properties without the need to provide the atomic composition,chemical stoichiometry,and cr...We present a full space inverse materials design(FSIMD)approach that fully automates the materials design for target physical properties without the need to provide the atomic composition,chemical stoichiometry,and crystal structure in advance.Here,we used density functional theory reference data to train a universal machine learning potential(UPot)and transfer learning to train a universal bulk modulus model(UBmod).Both UPot and UBmod were able to cover materials systems composed of any element among 42 elements.Interfaced with optimization algorithm and enhanced sampling,the FSIMD approach is applied to find the materials with the largest cohesive energy and the largest bulk modulus,respectively.NaCl-type ZrC was found to be the material with the largest cohesive energy.For bulk modulus,diamond was identified to have the largest value.The FSIMD approach is also applied to design materials with other multi-objective properties with accuracy limited principally by the amount,reliability,and diversity of the training data.The FSIMD approach provides a new way for inverse materials design with other functional properties for practical applications.展开更多
Relic gravitational waves(RGWs)from the early Universe carry crucial and fundamental cosmological information.Therefore,it is of extraordinary importance to investigate potential RGW signals in the data from observato...Relic gravitational waves(RGWs)from the early Universe carry crucial and fundamental cosmological information.Therefore,it is of extraordinary importance to investigate potential RGW signals in the data from observatories such as the LIGO-Virgo-KAGRA network.Here,focusing on typical RGWs from the inflation and the first-order phase transition(by sound waves and bubble collisions),effective and targeted deep learning neural networks are established to search for these RGW signals within the real LIGO data(O2,O3a and O3b).Through adjustment and adaptation processes,we develop suitable Convolutional Neural Networks(CNNs)to estimate the likelihood(characterized by quantitative values and distributions)that the focused RGW signals are present in the LIGO data.We find that if the constructed CNN properly estimates the parameters of the RGWs,it can determine with high accuracy(approximately 94%to 99%)whether the samples contain such RGW signals;otherwise,the likelihood provided by the CNN cannot be considered reliable.After testing a large amount of LIGO data,the findings show no evidence of RGWs from:1)inflation,2)sound waves,or 3)bubble collisions,as predicted by the focused theories.The results also provide upper limits of their GW spectral energy densities of h^(2)Ω_(gw)~10^(-5),respectively for parameter boundaries within 1)[β∈(-1.87,-1.85)×α∈(0.005,0.007)],2)[β/H_(pt)∈(0.02,0.16)×α∈(1,10)×T_(pt)∈(5*10^(9),10^(10))Gev],and 3)[β/H_(pt)∈(0.08,0.2)×α∈(1,10)×T_(pt)∈(5*10^(9),8*10^(10))Gev].In short,null results and upper limits are obtained,and the analysis suggests that our developed methods and neural networks to search for typical RGWs in the LIGO data are effective and reliable,providing a viable scheme for exploring possible RGWs from the early Universe and placing constraints on relevant cosmological theories.展开更多
Different from the concept of universal computation,the universality of a quantum neural network focuses on the ability to approximate arbitrary functions and is an important guarantee for effectiveness.However,conven...Different from the concept of universal computation,the universality of a quantum neural network focuses on the ability to approximate arbitrary functions and is an important guarantee for effectiveness.However,conventional approaches of constructing a universal quantum neural network may result in a huge quantum register that is challenging to implement due to noise on a near-term device.To address this,we propose a simple design of a duplication-free quantum neural network whose universality can be rigorously proven.Specifically,instead of using multiple duplicates of the quantum register,our method relies on a single quantum register combined with multiple activation functions to create nonlinearity and achieve universality.Accordingly,our proposal requires significantly fewer qubits with shallower circuits,and hence substantially reduces the resource overhead and the noise effect.In addition,simulations demonstrate that our universality design is able to achieve a better learning accuracy in the presence of noise,illustrating a great potential in solving larger-scale learning problems on near-term devices.展开更多
基金This work was supported by the National Research Foundation of Korea(NRF)grant funded by the Korea government(MSIT)(No.2022R1F1A1062953).
文摘Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity.Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process.However,it takes a lot of time and effort for researchers to use the existing activation function in their research.Therefore,in this paper,we propose a universal activation function(UA)so that researchers can easily create and apply various activation functions and improve the performance of neural networks.UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters.The famous Convolutional Neural Network(CNN)and benchmark datasetwere used to evaluate the experimental performance of the UA proposed in this study.We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which theUA is applied.In addition,we evaluated the performance of the new activation function generated by adjusting the hyperparameters of theUA.The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5%through the UA,although most of them showed similar performance to the traditional activation function.
文摘Due to the lack of large-scale emotion databases,it is hard to obtain comparable improvement in multimodal emotion recognition of the deep neural network by deep learning,which has made great progress in other areas.We use transfer learning to improve its performance with pretrained models on largescale data.Audio is encoded using deep speech recognition networks with 500 hours’speech and video is encoded using convolutional neural networks with over 110,000 images.The extracted audio and visual features are fed into Long Short-Term Memory to train models respectively.Logistic regression and ensemble method are performed in decision level fusion.The experiment results indicate that 1)audio features extracted from deep speech recognition networks achieve better performance than handcrafted audio features;2)the visual emotion recognition obtains better performance than audio emotion recognition;3)the ensemble method gets better performance than logistic regression and prior knowledge from micro-F1 value further improves the performance and robustness,achieving accuracy of 67.00%for“happy”,54.90%for“an?gry”,and 51.69%for“sad”.
文摘Smart Grids(SG)is a power system development concept that has received significant attention nationally.SG signifies real-time data for specific communication requirements.The best capabilities for monitoring and controlling the grid are essential to system stability.One of the most critical needs for smart-grid execution is fast,precise,and economically synchronized measurements,which are made feasible by Phasor Measurement Units(PMU).PMUs can pro-vide synchronized measurements and measure voltages as well as current phasors dynamically.PMUs utilize GPS time-stamping at Coordinated Universal Time(UTC)to capture electric phasors with great accuracy and precision.This research tends to Deep Learning(DL)advances to design a Residual Network(ResNet)model that can accurately identify and classify defects in grid-connected systems.As part of fault detection and probe,the proposed strategy uses a ResNet-50 tech-nique to evaluate real-time measurement data from geographically scattered PMUs.As a result of its excellent signal classification efficiency and ability to extract high-quality signal features,its fault diagnosis performance is excellent.Our results demonstrate that the proposed method is effective in detecting and classifying faults at sufficient time.The proposed approaches classify the fault type with a precision of 98.5%and an accuracy of 99.1%.The long-short-term memory(LSTM),Convolutional Neural Network(CNN),and CNN-LSTM algo-rithms are applied to compare the networks.Real-world data tends to evaluate these networks.
基金supported by the National Natural Science Foundation of China(Grant Nos.62272253 and 62272252)the Fundamental Research Funds for the Central Universities.It was also supported in part by the China Scholarship Council(CSC202406200085)the Innovation Project of Guangxi Graduate Education(YCBZ2024005).
文摘As various types of data grow explosively,largescale data storage,backup,and transmission become challenging,which motivates many researchers to propose efficient universal compression algorithms for multi-source data.In recent years,due to the emergence of hardware acceleration devices such as GPUs,TPUs,DPUs,and FPGAs,the performance bottleneck of neural networks(NN)has been overcome,making NN-based compression algorithms increasingly practical and popular.However,the research survey for the NN-based universal lossless compressors has not been conducted yet,and there is also a lack of unified evaluation metrics.To address the above problems,in this paper,we present a holistic survey as well as benchmark evaluations.Specifically,i)we thoroughly investigate NNbased lossless universal compression algorithms toward multisource data and classify them into 3 types:static pre-training,adaptive,and semi-adaptive.ii)We unify 19 evaluation metrics to comprehensively assess the compression effect,resource consumption,and model performance of compressors.iii)We conduct experiments more than 4600 CPU/GPU hours to evaluate 17 state-of-the-art compressors on 28 real-world datasets across data types of text,images,videos,audio,etc.iv)We also summarize the strengths and drawbacks of NNbased lossless data compressors and discuss promising research directions.We summarize the results as the NN-based Lossless Compressors Benchmark(NNLCB,See fahaihi.github.io/NNLCB website),which will be updated and maintained continuously in the future.
基金supported by National Natural Science Foundation of China(12402336,82201637,U20A2070,and 12025202)National High-Level Talent Project(YQR23069)+6 种基金Natural Science Foundation of Jiangsu Province(BK20230876)the Young Elite Scientist Sponsorship Program by CAST(YESS20210238)Forwardlooking layout projects(1002-ILB24009)Zhejang Provincial Medical and Health Technology Project(Grant No.2024KY246,2025KY180)Scientific Research Foundation of Hangzhou City University(No.J-202402)Open Research Fund of the State Key Laboratory of Brain-Machine Intelligence,Zhejiang University(Grant No.BMI2400025)Hangzhou Science and Technology Bureau.
文摘Light-field imaging has wide applications in various domains,including microscale life science imaging,mesoscale neuroimaging,and macroscale fluid dynamics imaging.The development of deep learning-based reconstruction methods has greatly facilitated high-resolution light-field image processing,however,current deep learning-based light-field reconstruction methods have predominantly concentrated on the microscale.Considering the multiscale imaging capacity of light-field technique,a network that can work over variant scales of light-field image reconstruction will significantly benefit the development of volumetric imaging.Unfortunately,to our knowledge,no one has reported a universal high-resolution light-field image reconstruction algorithm that is compatible with microscale,mesoscale,and macroscale.To fill this gap,we present a real-time and universal network(RTU-Net)to reconstruct high-resolution light-field images at any scale.RTU-Net,as the first network that works over multiscale light-field image reconstruction,employs an adaptive loss function based on generative adversarial theory and consequently exhibits strong generalization capability.We comprehensively assessed the performance of RTU-Net through the reconstruction of multiscale light-field images,including microscale tubulin and mitochondrion dataset,mesoscale synthetic mouse neuro dataset,and macroscale light-field particle imaging velocimetry dataset.The results indicated that RTU-Net has achieved real-time and high-resolution light-field image reconstruction for volume sizes ranging from 300μm×300μm×12μm to 25 mm×25 mm×25 mm,and demonstrated higher resolution when compared with recently reported light-field reconstruction networks.The high-resolution,strong robustness,high efficiency,and especially the general applicability of RTU-Net will significantly deepen our insight into high-resolution and volumetric imaging.
基金funding support by the National Key Research and Development Program of China(2020YFB1506400)the National Natural Science Foundation of China(11974257 and 12188101)+1 种基金Jiangsu Distinguished Young Talent Funding(BK20200003)Soochow Municipal Laboratory for low carbon technologies and industries.
文摘We present a full space inverse materials design(FSIMD)approach that fully automates the materials design for target physical properties without the need to provide the atomic composition,chemical stoichiometry,and crystal structure in advance.Here,we used density functional theory reference data to train a universal machine learning potential(UPot)and transfer learning to train a universal bulk modulus model(UBmod).Both UPot and UBmod were able to cover materials systems composed of any element among 42 elements.Interfaced with optimization algorithm and enhanced sampling,the FSIMD approach is applied to find the materials with the largest cohesive energy and the largest bulk modulus,respectively.NaCl-type ZrC was found to be the material with the largest cohesive energy.For bulk modulus,diamond was identified to have the largest value.The FSIMD approach is also applied to design materials with other multi-objective properties with accuracy limited principally by the amount,reliability,and diversity of the training data.The FSIMD approach provides a new way for inverse materials design with other functional properties for practical applications.
基金supported in part by the National Natural Science Foundation of China under Grant Nos.11605015,12347101 and 12147102the Natural Scienceof Chongqing under Grant No.cstc2020jcyjmsxm X0944the Research Funds for the Central Universities under Grant No.2022CDJXY-002。
文摘Relic gravitational waves(RGWs)from the early Universe carry crucial and fundamental cosmological information.Therefore,it is of extraordinary importance to investigate potential RGW signals in the data from observatories such as the LIGO-Virgo-KAGRA network.Here,focusing on typical RGWs from the inflation and the first-order phase transition(by sound waves and bubble collisions),effective and targeted deep learning neural networks are established to search for these RGW signals within the real LIGO data(O2,O3a and O3b).Through adjustment and adaptation processes,we develop suitable Convolutional Neural Networks(CNNs)to estimate the likelihood(characterized by quantitative values and distributions)that the focused RGW signals are present in the LIGO data.We find that if the constructed CNN properly estimates the parameters of the RGWs,it can determine with high accuracy(approximately 94%to 99%)whether the samples contain such RGW signals;otherwise,the likelihood provided by the CNN cannot be considered reliable.After testing a large amount of LIGO data,the findings show no evidence of RGWs from:1)inflation,2)sound waves,or 3)bubble collisions,as predicted by the focused theories.The results also provide upper limits of their GW spectral energy densities of h^(2)Ω_(gw)~10^(-5),respectively for parameter boundaries within 1)[β∈(-1.87,-1.85)×α∈(0.005,0.007)],2)[β/H_(pt)∈(0.02,0.16)×α∈(1,10)×T_(pt)∈(5*10^(9),10^(10))Gev],and 3)[β/H_(pt)∈(0.08,0.2)×α∈(1,10)×T_(pt)∈(5*10^(9),8*10^(10))Gev].In short,null results and upper limits are obtained,and the analysis suggests that our developed methods and neural networks to search for typical RGWs in the LIGO data are effective and reliable,providing a viable scheme for exploring possible RGWs from the early Universe and placing constraints on relevant cosmological theories.
基金supported by the National Key R&D Program of China(Grant No.2018YFA0306703)the National Natural Science Foundation of China(Grant No.92265208)。
文摘Different from the concept of universal computation,the universality of a quantum neural network focuses on the ability to approximate arbitrary functions and is an important guarantee for effectiveness.However,conventional approaches of constructing a universal quantum neural network may result in a huge quantum register that is challenging to implement due to noise on a near-term device.To address this,we propose a simple design of a duplication-free quantum neural network whose universality can be rigorously proven.Specifically,instead of using multiple duplicates of the quantum register,our method relies on a single quantum register combined with multiple activation functions to create nonlinearity and achieve universality.Accordingly,our proposal requires significantly fewer qubits with shallower circuits,and hence substantially reduces the resource overhead and the noise effect.In addition,simulations demonstrate that our universality design is able to achieve a better learning accuracy in the presence of noise,illustrating a great potential in solving larger-scale learning problems on near-term devices.
