Deep learning(DL),derived from the domain of Artificial Neural Networks(ANN),forms one of the most essential components of modern deep learning algorithms.DL segmentation models rely on layer-by-layer convolution-base...Deep learning(DL),derived from the domain of Artificial Neural Networks(ANN),forms one of the most essential components of modern deep learning algorithms.DL segmentation models rely on layer-by-layer convolution-based feature representation,guided by forward and backward propagation.Acritical aspect of this process is the selection of an appropriate activation function(AF)to ensure robustmodel learning.However,existing activation functions often fail to effectively address the vanishing gradient problem or are complicated by the need for manual parameter tuning.Most current research on activation function design focuses on classification tasks using natural image datasets such asMNIST,CIFAR-10,and CIFAR-100.To address this gap,this study proposesMed-ReLU,a novel activation function specifically designed for medical image segmentation.Med-ReLU prevents deep learning models fromsuffering dead neurons or vanishing gradient issues.It is a hybrid activation function that combines the properties of ReLU and Softsign.For positive inputs,Med-ReLU adopts the linear behavior of ReLU to avoid vanishing gradients,while for negative inputs,it exhibits the Softsign’s polynomial convergence,ensuring robust training and avoiding inactive neurons across the training set.The training performance and segmentation accuracy ofMed-ReLU have been thoroughly evaluated,demonstrating stable learning behavior and resistance to overfitting.It consistently outperforms state-of-the-art activation functions inmedical image segmentation tasks.Designed as a parameter-free function,Med-ReLU is simple to implement in complex deep learning architectures,and its effectiveness spans various neural network models and anomaly detection scenarios.展开更多
A vehicle engine cooling system is of utmost importance to ensure that the engine operates in a safe temperature range.In most radiators that are used to cool an engine,water serves as a cooling fluid.The performance ...A vehicle engine cooling system is of utmost importance to ensure that the engine operates in a safe temperature range.In most radiators that are used to cool an engine,water serves as a cooling fluid.The performance of a radiator in terms of heat transmission is significantly influenced by the incorporation of nanoparticles into the cooling water.Concentration and uniformity of nanoparticle distribution are the two major factors for the practical use of nanofluids.The shape and size of nanoparticles also have a great impact on the performance of heat transfer.Many researchers are investigating the impact of nanoparticles on heat transfer.This study aims to develop an artificial neural network(ANN)model for predicting the thermal conductivity of an ethylene glycol(EG)/waterbased crystalline nanocellulose(CNC)nanofluid for cooling internal combustion engine.The implementation of an artificial neural network considering different activation functions in the hidden layer is made to find the bestmodel for the cooling of an engine using the nanofluid.Accuracies of the model with different activation functions in artificial neural networks are analyzed for different nanofluid concentrations and temperatures.In artificial neural networks,Levenberg–Marquardt is an optimization approach used with activation functions,including Tansig and Logsig functions in the training phase.The findings of each training,testing,and validation phase are presented to demonstrate the network that provides the highest level of accuracy.The best result was obtained with Tansig,which has a correlation of 0.99903 and an error of 3.7959×10^(–8).It has also been noticed that the Logsig function can also be a good model due to its correlation of 0.99890 and an error of 4.9218×10^(–8).Thus ourANNwith Tansig and Logsig functions demonstrates a high correlation between the actual output and the predicted output.展开更多
The purpose of this study was to address the challenges in predicting and classifying accuracy in modeling Container Dwell Time (CDT) using Artificial Neural Networks (ANN). This objective was driven by the suboptimal...The purpose of this study was to address the challenges in predicting and classifying accuracy in modeling Container Dwell Time (CDT) using Artificial Neural Networks (ANN). This objective was driven by the suboptimal outcomes reported in previous studies and sought to apply an innovative approach to improve these results. To achieve this, the study applied the Fusion of Activation Functions (FAFs) to a substantial dataset. This dataset included 307,594 container records from the Port of Tema from 2014 to 2022, encompassing both import and transit containers. The RandomizedSearchCV algorithm from Python’s Scikit-learn library was utilized in the methodological approach to yield the optimal activation function for prediction accuracy. The results indicated that “ajaLT”, a fusion of the Logistic and Hyperbolic Tangent Activation Functions, provided the best prediction accuracy, reaching a high of 82%. Despite these encouraging findings, it’s crucial to recognize the study’s limitations. While Fusion of Activation Functions is a promising method, further evaluation is necessary across different container types and port operations to ascertain the broader applicability and generalizability of these findings. The original value of this study lies in its innovative application of FAFs to CDT. Unlike previous studies, this research evaluates the method based on prediction accuracy rather than training time. It opens new avenues for machine learning engineers and researchers in applying FAFs to enhance prediction accuracy in CDT modeling, contributing to a previously underexplored area.展开更多
Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of ...Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity.Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process.However,it takes a lot of time and effort for researchers to use the existing activation function in their research.Therefore,in this paper,we propose a universal activation function(UA)so that researchers can easily create and apply various activation functions and improve the performance of neural networks.UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters.The famous Convolutional Neural Network(CNN)and benchmark datasetwere used to evaluate the experimental performance of the UA proposed in this study.We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which theUA is applied.In addition,we evaluated the performance of the new activation function generated by adjusting the hyperparameters of theUA.The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5%through the UA,although most of them showed similar performance to the traditional activation function.展开更多
In this paper, the multistability issue is discussed for delayed complex-valued recurrent neural networks with discontinuous real-imaginary-type activation functions. Based on a fixed theorem and stability definition,...In this paper, the multistability issue is discussed for delayed complex-valued recurrent neural networks with discontinuous real-imaginary-type activation functions. Based on a fixed theorem and stability definition, sufficient criteria are established for the existence and stability of multiple equilibria of complex-valued recurrent neural networks. The number of stable equilibria is larger than that of real-valued recurrent neural networks, which can be used to achieve high-capacity associative memories. One numerical example is provided to show the effectiveness and superiority of the presented results.展开更多
The stability of a periodic oscillation and the global exponential class of recurrent neural networks with non-monotone activation functions and time-varying delays are analyzed. For two sets of activation functions, ...The stability of a periodic oscillation and the global exponential class of recurrent neural networks with non-monotone activation functions and time-varying delays are analyzed. For two sets of activation functions, some algebraic criteria for ascertaining global exponential periodicity and global exponential stability of the class of recurrent neural networks are derived by using the comparison principle and the theory of monotone operator. These conditions are easy to check in terms of system parameters. In addition, we provide a new and efficacious method for the qualitative analysis of various neural networks.展开更多
Activation functions play an essential role in converting the output of the artificial neural network into nonlinear results,since without this nonlinearity,the results of the network will be less accurate.Nonlinearity...Activation functions play an essential role in converting the output of the artificial neural network into nonlinear results,since without this nonlinearity,the results of the network will be less accurate.Nonlinearity is the mission of all nonlinear functions,except for polynomials.The activation function must be dif-ferentiable for backpropagation learning.This study’s objective is to determine the best activation functions for the approximation of each fractal image.Different results have been attained using Matlab and Visual Basic programs,which indi-cate that the bounded function is more helpful than other functions.The non-lin-earity of the activation function is important when using neural networks for coding fractal images because the coefficients of the Iterated Function System are different according to the different types of fractals.The most commonly cho-sen activation function is the sigmoidal function,which produces a positive value.Other functions,such as tansh or arctan,whose values can be positive or negative depending on the network input,tend to train neural networks faster.The coding speed of the fractal image is different depending on the appropriate activation function chosen for each fractal shape.In this paper,we have provided the appro-priate activation functions for each type of system of iterated functions that help the network to identify the transactions of the system.展开更多
This paper describes our implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset—the Modified National Institute of Standards and Te...This paper describes our implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset—the Modified National Institute of Standards and Technology (MNIST) database. We also propose a novel hardware-friendly activation function called the dynamic Rectifid Linear Unit (ReLU)—D-ReLU function that achieves higher performance than traditional activation functions at no cost to accuracy. We built a 2-layer online training multilayer perceptron (MLP) neural network on an FPGA with varying data width. Reducing the data width from 8 to 4 bits only reduces prediction accuracy by 11%, but the FPGA area decreases by 41%. Compared to networks that use the sigmoid functions, our proposed D-ReLU function uses 24% - 41% less area with no loss to prediction accuracy. Further reducing the data width of the 3-layer networks from 8 to 4 bits, the prediction accuracies only decrease by 3% - 5%, with area being reduced by 9% - 28%. Moreover, FPGA solutions have 29 times faster execution time, even despite running at a 60× lower clock rate. Thus, FPGA implementations of neural networks offer a high-performance, low power alternative to traditional software methods, and our novel D-ReLU activation function offers additional improvements to performance and power saving.展开更多
In this work,an H_(∞)/passive-based secure synchronization control problem is investigated for continuous-time semi-Markov neural networks subject to hybrid attacks,in which hybrid attacks are the combinations of den...In this work,an H_(∞)/passive-based secure synchronization control problem is investigated for continuous-time semi-Markov neural networks subject to hybrid attacks,in which hybrid attacks are the combinations of denial-of-service attacks and deception attacks,and they are described by two groups of independent Bernoulli distributions.On this foundation,via the Lyapunov stability theory and linear matrix inequality technology,the H_(∞)/passive-based performance criteria for semi-Markov jump neural networks are obtained.Additionally,an activation function division approach for neural networks is adopted to further reduce the conservatism of the criteria.Finally,a simulation example is provided to verify the validity and feasibility of the proposed method.展开更多
Binary neural networks(BNNs)show promising utilization in cost and power-restricted domains such as edge devices and mobile systems.This is due to its significantly less computation and storage demand,but at the cost ...Binary neural networks(BNNs)show promising utilization in cost and power-restricted domains such as edge devices and mobile systems.This is due to its significantly less computation and storage demand,but at the cost of degraded performance.To close the accuracy gap,in this paper we propose to add a complementary activation function(AF)ahead of the sign based binarization,and rely on the genetic algorithm(GA)to automatically search for the ideal AFs.These AFs can help extract extra information from the input data in the forward pass,while allowing improved gradient approximation in the backward pass.Fifteen novel AFs are identified through our GA-based search,while most of them show improved performance(up to 2.54%on ImageNet)when testing on different datasets and network models.Interestingly,periodic functions are identified as a key component for most of the discovered AFs,which rarely exist in human designed AFs.Our method offers a novel approach for designing general and application-specific BNN architecture.GAAF will be released on GitHub.展开更多
Automatic Speech Recognition(ASR)is the process of mapping an acoustic speech signal into a human readable text format.Traditional systems exploit the Acoustic Component of ASR using the Gaussian Mixture Model-Hidden ...Automatic Speech Recognition(ASR)is the process of mapping an acoustic speech signal into a human readable text format.Traditional systems exploit the Acoustic Component of ASR using the Gaussian Mixture Model-Hidden Markov Model(GMM-HMM)approach.Deep NeuralNetwork(DNN)opens up new possibilities to overcome the shortcomings of conventional statistical algorithms.Recent studies modeled the acoustic component of ASR system using DNN in the so called hybrid DNN-HMM approach.In the context of activation functions used to model the non-linearity in DNN,Rectified Linear Units(ReLU)and maxout units are mostly used in ASR systems.This paper concentrates on the acoustic component of a hybrid DNN-HMM system by proposing an efficient activation function for the DNN network.Inspired by previous works,euclidean norm activation function is proposed to model the non-linearity of the DNN network.Such non-linearity is shown to belong to the family of Piecewise Linear(PWL)functions having distinct features.These functions can capture deep hierarchical features of the pattern.The relevance of the proposal is examined in depth both theoretically and experimentally.The performance of the developed ASR system is evaluated in terms of Phone Error Rate(PER)using TIMIT database.Experimental results achieve a relative increase in performance by using the proposed function over conventional activation functions.展开更多
Physics-informed neural networks(PINNs)are known to suffer from optimization difficulty.In this work,we reveal the connection between the optimization difficulty of PINNs and activation functions.Specifically,we show ...Physics-informed neural networks(PINNs)are known to suffer from optimization difficulty.In this work,we reveal the connection between the optimization difficulty of PINNs and activation functions.Specifically,we show that PINNs exhibit high sensitivity to activation functions when solving PDEs with distinct properties.Existing works usually choose activation functions by inefficient trial-and-error.To avoid the inefficient manual selection and to alleviate the optimization difficulty of PINNs,we introduce adaptive activation functions to search for the optimal function when solving different problems.We compare different adaptive activation functions and discuss their limitations in the context of PINNs.Furthermore,we propose to tailor the idea of learning combinations of candidate activation functions to the PINNs optimization,which has a higher requirement for the smoothness and diversity on learned functions.This is achieved by removing activation functions which cannot provide higher-order derivatives from the candidate set and incorporating elementary functions with different properties according to our prior knowledge about the PDE at hand.We further enhance the search space with adaptive slopes.The proposed adaptive activation function can be used to solve different PDE systems in an interpretable way.Its effectiveness is demonstrated on a series of benchmarks.Code is available at https://github.com/LeapLabTHU/AdaAFforPINNs.展开更多
Purpose–The purpose of this paper is to develop a method for the existence,uniqueness and globally robust stability of the equilibrium point for Cohen–Grossberg neural networks with time-varying delays,continuous di...Purpose–The purpose of this paper is to develop a method for the existence,uniqueness and globally robust stability of the equilibrium point for Cohen–Grossberg neural networks with time-varying delays,continuous distributed delays and a kind of discontinuous activation functions.Design/methodology/approach–Basedonthe Leray–Schauderalternativetheoremand chainrule,by using a novel integral inequality dealing with monotone non-decreasing function,the authors obtain a delay-dependent sufficient condition with less conservativeness for robust stability of considered neural networks.Findings–Itturns out thattheauthors’delay-dependent sufficientcondition canbeformed intermsof linear matrix inequalities conditions.Two examples show the effectiveness of the obtained results.Originality/value–The novelty of the proposed approach lies in dealing with a new kind of discontinuous activation functions by using the Leray–Schauder alternative theorem,chain rule and a novel integral inequality on monotone non-decreasing function.展开更多
This paper discusses the general decay synchronization problem for a class of fuzzy competitive neural networks with time-varying delays and discontinuous activation functions. Firstly, based on the concept of Filippo...This paper discusses the general decay synchronization problem for a class of fuzzy competitive neural networks with time-varying delays and discontinuous activation functions. Firstly, based on the concept of Filippov solutions for right-hand discontinuous systems, some sufficient conditions for general decay synchronization of the considered system are obtained via designing a nonlinear feedback controller and applying discontinuous differential equation theory, Lyapunov functional methods and some inequality techniques. Finally, one numerical example is given to verify the effectiveness of the proposed theoretical results. The general decay synchronization considered in this article can better estimate the convergence rate of the system, and the exponential synchronization and polynomial synchronization can be seen as its special cases.展开更多
Background:Excessive heat exposure can lead to hyperthermia in humans,which impairs physical performance and disrupts cognitive function.While heat is a known physiological stressor,it is unclear how severe heat stres...Background:Excessive heat exposure can lead to hyperthermia in humans,which impairs physical performance and disrupts cognitive function.While heat is a known physiological stressor,it is unclear how severe heat stress affects brain physiology and function.Methods:Eleven healthy participants were subjected to heat stress from prolonged exercise or warm water immersion until their rectal temperatures(T_(re))attained 39.5℃,inducing exertional or passive hyperthermia,respectively.In a separate trial,blended ice was ingested before and during exercise as a cooling strategy.Data were compared to a control condition with seated rest(normothermic).Brain temperature(T_(br)),cerebral perfusion,and task-based brain activity were assessed using magnetic resonance imaging techniques.Results:T_(br)in motor cortex was found to be tightly regulated at rest(37.3℃±0.4℃(mean±SD))despite fluctuations in T_(re).With the development of hyperthermia,T_(br)increases and dovetails with the rising T_(re).Bilateral motor cortical activity was suppressed during high-intensity plantarflexion tasks,implying a reduced central motor drive in hyperthermic participants(T_(re)=38.5℃±0.1℃).Global gray matter perfusion and regional perfusion in sensorimotor cortex were reduced with passive hyperthermia.Executive function was poorer under a passive hyperthermic state,and this could relate to compromised visual processing as indicated by the reduced activation of left lateral-occipital cortex.Conversely,ingestion of blended ice before and during exercise alleviated the rise in both T_(re)and T_(bc)and mitigated heat-related neural perturbations.Conclusion:Severe heat exposure elevates T_(br),disrupts motor cortical activity and executive function,and this can lead to impairment of physical and cognitive performance.展开更多
Density functional theory calculations were carried out to explore the potential energy surface（PES） associated with the gas-phase reaction of Ni L2（L=SO3CH3） with acetone. The geometries and energies of the react...Density functional theory calculations were carried out to explore the potential energy surface（PES） associated with the gas-phase reaction of Ni L2（L=SO3CH3） with acetone. The geometries and energies of the reactants, intermediates, products and transition states of the triplet ground potential energy surfaces of [Ni, O, C2, H4] were obtained at the B3LYP/6-311＋＋G（d,p） levels in C,H,O atoms and B3LYP/ Lanl2 dz in Ni atom. It was found through our calculations that the decabonylation of acetaldehyde contains four steps including encounter complexation, C-C activation, aldehyde H-shift and nonreactive dissociation. The results revealed that C-C activation induced by Ni L2（L=SO3CH3） led to the decarbonylation of acetaldehyde.展开更多
In the field of image processing,the analysis of Synthetic Aperture Radar(SAR)images is crucial due to its broad range of applications.However,SAR images are often affected by coherent speckle noise,which significantl...In the field of image processing,the analysis of Synthetic Aperture Radar(SAR)images is crucial due to its broad range of applications.However,SAR images are often affected by coherent speckle noise,which significantly degrades image quality.Traditional denoising methods,typically based on filter techniques,often face challenges related to inefficiency and limited adaptability.To address these limitations,this study proposes a novel SAR image denoising algorithm based on an enhanced residual network architecture,with the objective of enhancing the utility of SAR imagery in complex electromagnetic environments.The proposed algorithm integrates residual network modules,which directly process the noisy input images to generate denoised outputs.This approach not only reduces computational complexity but also mitigates the difficulties associated with model training.By combining the Transformer module with the residual block,the algorithm enhances the network's ability to extract global features,offering superior feature extraction capabilities compared to CNN-based residual modules.Additionally,the algorithm employs the adaptive activation function Meta-ACON,which dynamically adjusts the activation patterns of neurons,thereby improving the network's feature extraction efficiency.The effectiveness of the proposed denoising method is empirically validated using real SAR images from the RSOD dataset.The proposed algorithm exhibits remarkable performance in terms of EPI,SSIM,and ENL,while achieving a substantial enhancement in PSNR when compared to traditional and deep learning-based algorithms.The PSNR performance is enhanced by over twofold.Moreover,the evaluation of the MSTAR SAR dataset substantiates the algorithm's robustness and applicability in SAR denoising tasks,with a PSNR of 25.2021 being attained.These findings underscore the efficacy of the proposed algorithm in mitigating speckle noise while preserving critical features in SAR imagery,thereby enhancing its quality and usability in practical scenarios.展开更多
To improve the performance of multilayer perceptron(MLP)neural networks activated by conventional activation functions,this paper presents a new MLP activated by univariate Gaussian radial basis functions(RBFs)with ad...To improve the performance of multilayer perceptron(MLP)neural networks activated by conventional activation functions,this paper presents a new MLP activated by univariate Gaussian radial basis functions(RBFs)with adaptive centers and widths,which is composed of more than one hidden layer.In the hidden layer of the RBF-activated MLP network(MLPRBF),the outputs of the preceding layer are first linearly transformed and then fed into the univariate Gaussian RBF,which exploits the highly nonlinear property of RBF.Adaptive RBFs might address the issues of saturated outputs,low sensitivity,and vanishing gradients in MLPs activated by other prevailing nonlinear functions.Finally,we apply four MLP networks with the rectified linear unit(ReLU),sigmoid function(sigmoid),hyperbolic tangent function(tanh),and Gaussian RBF as the activation functions to approximate the one-dimensional(1D)sinusoidal function,the analytical solution of viscous Burgers’equation,and the two-dimensional(2D)steady lid-driven cavity flows.Using the same network structure,MLP-RBF generally predicts more accurately and converges faster than the other threeMLPs.MLP-RBF using less hidden layers and/or neurons per layer can yield comparable or even higher approximation accuracy than other MLPs equipped with more layers or neurons.展开更多
Machine Learning(ML)and Deep Learning(DL)technologies are revolutionizing the medical domain,especially with Electrocardiogram(ECG),by providing new tools and techniques for diagnosing,treating,and preventing diseases...Machine Learning(ML)and Deep Learning(DL)technologies are revolutionizing the medical domain,especially with Electrocardiogram(ECG),by providing new tools and techniques for diagnosing,treating,and preventing diseases.However,DL architectures are computationally more demanding.In recent years,researchers have focused on combining the computationally less intensive portion of the DL architectures with ML approaches,say for example,combining the convolutional layer blocks of Convolution Neural Networks(CNNs)into ML algorithms such as Extreme Gradient Boosting(XGBoost)and K-Nearest Neighbor(KNN)resulting in CNN-XGBoost and CNN-KNN,respectively.However,these approaches are homogenous in the sense that they use a fixed Activation Function(AFs)in the sequence of convolution and pooling layers,thereby limiting the ability to capture unique features.Since various AFs are readily available and each could capture unique features,we propose a Convolutionbased Heterogeneous Activation Facility(CHAF)which uses multiple AFs in the convolution layer blocks,one for each block,with a motivation of extracting features in a better manner to improve the accuracy.The proposed CHAF approach is validated on PTB and shown to outperform the homogeneous approaches such as CNN-KNN and CNN-XGBoost.For PTB dataset,proposed CHAF-KNN has an accuracy of 99.55%and an F1 score of 99.68%in just 0.008 s,outperforming the state-of-the-art CNN-XGBoost which has an accuracy of 99.38%and an F1 score of 99.32%in 1.23 s.To validate the generality of the proposed CHAF,experiments were repeated on MIT-BIH dataset,and the proposed CHAF-KNN is shown to outperform CNN-KNN and CNN-XGBoost.展开更多
Objective:To conduct a scoping review on the application status of the Functional Activity Score(FAS)in postoperative active pain management in China,providing a reference for its standardized and normative promotion....Objective:To conduct a scoping review on the application status of the Functional Activity Score(FAS)in postoperative active pain management in China,providing a reference for its standardized and normative promotion.Methods:Computerized searches of Chinese and English databases were performed to collect studies published by Chinese scholars from 2005 to July 2025 on the application of FAS in postoperative active pain management.After strict screening,the basic characteristics,application fields,assessment models,evaluation timing,types of functional activities,and clinical outcomes of the included literature were systematically analyzed.Results:A total of 18 studies were included,involving surgical types such as thoracic surgery,general surgery,and orthopedics.All studies adopted FAS combined with the Numeric Rating Scale(NRS)for assessment,with evaluation timing mostly concentrated within 72 hours postoperatively.The selected functional activities primarily included respiration-related and limb movements.Evaluation indicators covered pain control,functional recovery,complications,adverse events,patient experience,and tool assessment,with most studies reporting positive outcomes.Conclusion:FAS can effectively enhance pain control and promote functional recovery in postoperative active pain management in China,demonstrating high clinical value.However,existing studies exhibit inconsistencies in assessment criteria,selection of activity types,and research quality.展开更多
基金The researchers would like to thank the Deanship of Graduate Studies and Scientific Research at Qassim University for financial support(QU-APC-2025).
文摘Deep learning(DL),derived from the domain of Artificial Neural Networks(ANN),forms one of the most essential components of modern deep learning algorithms.DL segmentation models rely on layer-by-layer convolution-based feature representation,guided by forward and backward propagation.Acritical aspect of this process is the selection of an appropriate activation function(AF)to ensure robustmodel learning.However,existing activation functions often fail to effectively address the vanishing gradient problem or are complicated by the need for manual parameter tuning.Most current research on activation function design focuses on classification tasks using natural image datasets such asMNIST,CIFAR-10,and CIFAR-100.To address this gap,this study proposesMed-ReLU,a novel activation function specifically designed for medical image segmentation.Med-ReLU prevents deep learning models fromsuffering dead neurons or vanishing gradient issues.It is a hybrid activation function that combines the properties of ReLU and Softsign.For positive inputs,Med-ReLU adopts the linear behavior of ReLU to avoid vanishing gradients,while for negative inputs,it exhibits the Softsign’s polynomial convergence,ensuring robust training and avoiding inactive neurons across the training set.The training performance and segmentation accuracy ofMed-ReLU have been thoroughly evaluated,demonstrating stable learning behavior and resistance to overfitting.It consistently outperforms state-of-the-art activation functions inmedical image segmentation tasks.Designed as a parameter-free function,Med-ReLU is simple to implement in complex deep learning architectures,and its effectiveness spans various neural network models and anomaly detection scenarios.
基金supported by the International Publication Research Grant No.RDU223301.
文摘A vehicle engine cooling system is of utmost importance to ensure that the engine operates in a safe temperature range.In most radiators that are used to cool an engine,water serves as a cooling fluid.The performance of a radiator in terms of heat transmission is significantly influenced by the incorporation of nanoparticles into the cooling water.Concentration and uniformity of nanoparticle distribution are the two major factors for the practical use of nanofluids.The shape and size of nanoparticles also have a great impact on the performance of heat transfer.Many researchers are investigating the impact of nanoparticles on heat transfer.This study aims to develop an artificial neural network(ANN)model for predicting the thermal conductivity of an ethylene glycol(EG)/waterbased crystalline nanocellulose(CNC)nanofluid for cooling internal combustion engine.The implementation of an artificial neural network considering different activation functions in the hidden layer is made to find the bestmodel for the cooling of an engine using the nanofluid.Accuracies of the model with different activation functions in artificial neural networks are analyzed for different nanofluid concentrations and temperatures.In artificial neural networks,Levenberg–Marquardt is an optimization approach used with activation functions,including Tansig and Logsig functions in the training phase.The findings of each training,testing,and validation phase are presented to demonstrate the network that provides the highest level of accuracy.The best result was obtained with Tansig,which has a correlation of 0.99903 and an error of 3.7959×10^(–8).It has also been noticed that the Logsig function can also be a good model due to its correlation of 0.99890 and an error of 4.9218×10^(–8).Thus ourANNwith Tansig and Logsig functions demonstrates a high correlation between the actual output and the predicted output.
文摘The purpose of this study was to address the challenges in predicting and classifying accuracy in modeling Container Dwell Time (CDT) using Artificial Neural Networks (ANN). This objective was driven by the suboptimal outcomes reported in previous studies and sought to apply an innovative approach to improve these results. To achieve this, the study applied the Fusion of Activation Functions (FAFs) to a substantial dataset. This dataset included 307,594 container records from the Port of Tema from 2014 to 2022, encompassing both import and transit containers. The RandomizedSearchCV algorithm from Python’s Scikit-learn library was utilized in the methodological approach to yield the optimal activation function for prediction accuracy. The results indicated that “ajaLT”, a fusion of the Logistic and Hyperbolic Tangent Activation Functions, provided the best prediction accuracy, reaching a high of 82%. Despite these encouraging findings, it’s crucial to recognize the study’s limitations. While Fusion of Activation Functions is a promising method, further evaluation is necessary across different container types and port operations to ascertain the broader applicability and generalizability of these findings. The original value of this study lies in its innovative application of FAFs to CDT. Unlike previous studies, this research evaluates the method based on prediction accuracy rather than training time. It opens new avenues for machine learning engineers and researchers in applying FAFs to enhance prediction accuracy in CDT modeling, contributing to a previously underexplored area.
基金This work was supported by the National Research Foundation of Korea(NRF)grant funded by the Korea government(MSIT)(No.2022R1F1A1062953).
文摘Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity.Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process.However,it takes a lot of time and effort for researchers to use the existing activation function in their research.Therefore,in this paper,we propose a universal activation function(UA)so that researchers can easily create and apply various activation functions and improve the performance of neural networks.UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters.The famous Convolutional Neural Network(CNN)and benchmark datasetwere used to evaluate the experimental performance of the UA proposed in this study.We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which theUA is applied.In addition,we evaluated the performance of the new activation function generated by adjusting the hyperparameters of theUA.The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5%through the UA,although most of them showed similar performance to the traditional activation function.
基金Project supported by the National Natural Science Foundation of China(Grant Nos.61374094 and 61503338)the Natural Science Foundation of Zhejiang Province,China(Grant No.LQ15F030005)
文摘In this paper, the multistability issue is discussed for delayed complex-valued recurrent neural networks with discontinuous real-imaginary-type activation functions. Based on a fixed theorem and stability definition, sufficient criteria are established for the existence and stability of multiple equilibria of complex-valued recurrent neural networks. The number of stable equilibria is larger than that of real-valued recurrent neural networks, which can be used to achieve high-capacity associative memories. One numerical example is provided to show the effectiveness and superiority of the presented results.
基金Supported by the Natural Science Foundation of Hubei Province (2007ABA124)the Youth Project Foundation of Hubei Province Education Department (Q200722001)the Major Foundation of Hubei Province Education Department (D200722002)
文摘The stability of a periodic oscillation and the global exponential class of recurrent neural networks with non-monotone activation functions and time-varying delays are analyzed. For two sets of activation functions, some algebraic criteria for ascertaining global exponential periodicity and global exponential stability of the class of recurrent neural networks are derived by using the comparison principle and the theory of monotone operator. These conditions are easy to check in terms of system parameters. In addition, we provide a new and efficacious method for the qualitative analysis of various neural networks.
文摘Activation functions play an essential role in converting the output of the artificial neural network into nonlinear results,since without this nonlinearity,the results of the network will be less accurate.Nonlinearity is the mission of all nonlinear functions,except for polynomials.The activation function must be dif-ferentiable for backpropagation learning.This study’s objective is to determine the best activation functions for the approximation of each fractal image.Different results have been attained using Matlab and Visual Basic programs,which indi-cate that the bounded function is more helpful than other functions.The non-lin-earity of the activation function is important when using neural networks for coding fractal images because the coefficients of the Iterated Function System are different according to the different types of fractals.The most commonly cho-sen activation function is the sigmoidal function,which produces a positive value.Other functions,such as tansh or arctan,whose values can be positive or negative depending on the network input,tend to train neural networks faster.The coding speed of the fractal image is different depending on the appropriate activation function chosen for each fractal shape.In this paper,we have provided the appro-priate activation functions for each type of system of iterated functions that help the network to identify the transactions of the system.
文摘This paper describes our implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset—the Modified National Institute of Standards and Technology (MNIST) database. We also propose a novel hardware-friendly activation function called the dynamic Rectifid Linear Unit (ReLU)—D-ReLU function that achieves higher performance than traditional activation functions at no cost to accuracy. We built a 2-layer online training multilayer perceptron (MLP) neural network on an FPGA with varying data width. Reducing the data width from 8 to 4 bits only reduces prediction accuracy by 11%, but the FPGA area decreases by 41%. Compared to networks that use the sigmoid functions, our proposed D-ReLU function uses 24% - 41% less area with no loss to prediction accuracy. Further reducing the data width of the 3-layer networks from 8 to 4 bits, the prediction accuracies only decrease by 3% - 5%, with area being reduced by 9% - 28%. Moreover, FPGA solutions have 29 times faster execution time, even despite running at a 60× lower clock rate. Thus, FPGA implementations of neural networks offer a high-performance, low power alternative to traditional software methods, and our novel D-ReLU activation function offers additional improvements to performance and power saving.
基金supported by the National Natural Science Foundation of China under Grant Nos.62103005,62173001,and 62273006the Natural Science Foundation of Anhui Provincial Natural Science Foundation under Grant No.2108085QF276+3 种基金the Natural Science Foundation for Distinguished Young Scholars of Higher Education Institutions of Anhui Province under Grant No.2022AH020034the Natural Science Foundation for Excellent Young Scholars of Higher Education Institutions of Anhui Province under Grant No.2022AH030049,2023AH030030,2022AH030049the Major Technologies Research and Development Special Program of Anhui Province under Grant No.202003a05020001the Key Research and Development Projects of Anhui Province under Grant No.202104a05020015。
文摘In this work,an H_(∞)/passive-based secure synchronization control problem is investigated for continuous-time semi-Markov neural networks subject to hybrid attacks,in which hybrid attacks are the combinations of denial-of-service attacks and deception attacks,and they are described by two groups of independent Bernoulli distributions.On this foundation,via the Lyapunov stability theory and linear matrix inequality technology,the H_(∞)/passive-based performance criteria for semi-Markov jump neural networks are obtained.Additionally,an activation function division approach for neural networks is adopted to further reduce the conservatism of the criteria.Finally,a simulation example is provided to verify the validity and feasibility of the proposed method.
文摘Binary neural networks(BNNs)show promising utilization in cost and power-restricted domains such as edge devices and mobile systems.This is due to its significantly less computation and storage demand,but at the cost of degraded performance.To close the accuracy gap,in this paper we propose to add a complementary activation function(AF)ahead of the sign based binarization,and rely on the genetic algorithm(GA)to automatically search for the ideal AFs.These AFs can help extract extra information from the input data in the forward pass,while allowing improved gradient approximation in the backward pass.Fifteen novel AFs are identified through our GA-based search,while most of them show improved performance(up to 2.54%on ImageNet)when testing on different datasets and network models.Interestingly,periodic functions are identified as a key component for most of the discovered AFs,which rarely exist in human designed AFs.Our method offers a novel approach for designing general and application-specific BNN architecture.GAAF will be released on GitHub.
基金This work was an outcome of the R&D work undertaken project under the Visvesvaraya PhD Scheme of Ministry of Electronics&Information TechnologyGovernment of India,being implemented by Digital India Corporation.
文摘Automatic Speech Recognition(ASR)is the process of mapping an acoustic speech signal into a human readable text format.Traditional systems exploit the Acoustic Component of ASR using the Gaussian Mixture Model-Hidden Markov Model(GMM-HMM)approach.Deep NeuralNetwork(DNN)opens up new possibilities to overcome the shortcomings of conventional statistical algorithms.Recent studies modeled the acoustic component of ASR system using DNN in the so called hybrid DNN-HMM approach.In the context of activation functions used to model the non-linearity in DNN,Rectified Linear Units(ReLU)and maxout units are mostly used in ASR systems.This paper concentrates on the acoustic component of a hybrid DNN-HMM system by proposing an efficient activation function for the DNN network.Inspired by previous works,euclidean norm activation function is proposed to model the non-linearity of the DNN network.Such non-linearity is shown to belong to the family of Piecewise Linear(PWL)functions having distinct features.These functions can capture deep hierarchical features of the pattern.The relevance of the proposal is examined in depth both theoretically and experimentally.The performance of the developed ASR system is evaluated in terms of Phone Error Rate(PER)using TIMIT database.Experimental results achieve a relative increase in performance by using the proposed function over conventional activation functions.
基金supported in part by the National Natural Science Foundation of China under Grants 62276150the Guoqiang Institute of Tsinghua University.
文摘Physics-informed neural networks(PINNs)are known to suffer from optimization difficulty.In this work,we reveal the connection between the optimization difficulty of PINNs and activation functions.Specifically,we show that PINNs exhibit high sensitivity to activation functions when solving PDEs with distinct properties.Existing works usually choose activation functions by inefficient trial-and-error.To avoid the inefficient manual selection and to alleviate the optimization difficulty of PINNs,we introduce adaptive activation functions to search for the optimal function when solving different problems.We compare different adaptive activation functions and discuss their limitations in the context of PINNs.Furthermore,we propose to tailor the idea of learning combinations of candidate activation functions to the PINNs optimization,which has a higher requirement for the smoothness and diversity on learned functions.This is achieved by removing activation functions which cannot provide higher-order derivatives from the candidate set and incorporating elementary functions with different properties according to our prior knowledge about the PDE at hand.We further enhance the search space with adaptive slopes.The proposed adaptive activation function can be used to solve different PDE systems in an interpretable way.Its effectiveness is demonstrated on a series of benchmarks.Code is available at https://github.com/LeapLabTHU/AdaAFforPINNs.
基金supported by the National Natural Science Foundation of China No.61273022the Research Foundation of Department of Education of Liaoning Province No.JDL2017031.
文摘Purpose–The purpose of this paper is to develop a method for the existence,uniqueness and globally robust stability of the equilibrium point for Cohen–Grossberg neural networks with time-varying delays,continuous distributed delays and a kind of discontinuous activation functions.Design/methodology/approach–Basedonthe Leray–Schauderalternativetheoremand chainrule,by using a novel integral inequality dealing with monotone non-decreasing function,the authors obtain a delay-dependent sufficient condition with less conservativeness for robust stability of considered neural networks.Findings–Itturns out thattheauthors’delay-dependent sufficientcondition canbeformed intermsof linear matrix inequalities conditions.Two examples show the effectiveness of the obtained results.Originality/value–The novelty of the proposed approach lies in dealing with a new kind of discontinuous activation functions by using the Leray–Schauder alternative theorem,chain rule and a novel integral inequality on monotone non-decreasing function.
文摘This paper discusses the general decay synchronization problem for a class of fuzzy competitive neural networks with time-varying delays and discontinuous activation functions. Firstly, based on the concept of Filippov solutions for right-hand discontinuous systems, some sufficient conditions for general decay synchronization of the considered system are obtained via designing a nonlinear feedback controller and applying discontinuous differential equation theory, Lyapunov functional methods and some inequality techniques. Finally, one numerical example is given to verify the effectiveness of the proposed theoretical results. The general decay synchronization considered in this article can better estimate the convergence rate of the system, and the exponential synchronization and polynomial synchronization can be seen as its special cases.
基金supported by Defence Innovative Research Program(DIRP)Grant(PA No.9015102335)from Defence Research&Technology Office,Ministry of Defence,Singapore。
文摘Background:Excessive heat exposure can lead to hyperthermia in humans,which impairs physical performance and disrupts cognitive function.While heat is a known physiological stressor,it is unclear how severe heat stress affects brain physiology and function.Methods:Eleven healthy participants were subjected to heat stress from prolonged exercise or warm water immersion until their rectal temperatures(T_(re))attained 39.5℃,inducing exertional or passive hyperthermia,respectively.In a separate trial,blended ice was ingested before and during exercise as a cooling strategy.Data were compared to a control condition with seated rest(normothermic).Brain temperature(T_(br)),cerebral perfusion,and task-based brain activity were assessed using magnetic resonance imaging techniques.Results:T_(br)in motor cortex was found to be tightly regulated at rest(37.3℃±0.4℃(mean±SD))despite fluctuations in T_(re).With the development of hyperthermia,T_(br)increases and dovetails with the rising T_(re).Bilateral motor cortical activity was suppressed during high-intensity plantarflexion tasks,implying a reduced central motor drive in hyperthermic participants(T_(re)=38.5℃±0.1℃).Global gray matter perfusion and regional perfusion in sensorimotor cortex were reduced with passive hyperthermia.Executive function was poorer under a passive hyperthermic state,and this could relate to compromised visual processing as indicated by the reduced activation of left lateral-occipital cortex.Conversely,ingestion of blended ice before and during exercise alleviated the rise in both T_(re)and T_(bc)and mitigated heat-related neural perturbations.Conclusion:Severe heat exposure elevates T_(br),disrupts motor cortical activity and executive function,and this can lead to impairment of physical and cognitive performance.
基金Funded by the National Natural Science Foundation of China(No.51174179)
文摘Density functional theory calculations were carried out to explore the potential energy surface（PES） associated with the gas-phase reaction of Ni L2（L=SO3CH3） with acetone. The geometries and energies of the reactants, intermediates, products and transition states of the triplet ground potential energy surfaces of [Ni, O, C2, H4] were obtained at the B3LYP/6-311＋＋G（d,p） levels in C,H,O atoms and B3LYP/ Lanl2 dz in Ni atom. It was found through our calculations that the decabonylation of acetaldehyde contains four steps including encounter complexation, C-C activation, aldehyde H-shift and nonreactive dissociation. The results revealed that C-C activation induced by Ni L2（L=SO3CH3） led to the decarbonylation of acetaldehyde.
文摘In the field of image processing,the analysis of Synthetic Aperture Radar(SAR)images is crucial due to its broad range of applications.However,SAR images are often affected by coherent speckle noise,which significantly degrades image quality.Traditional denoising methods,typically based on filter techniques,often face challenges related to inefficiency and limited adaptability.To address these limitations,this study proposes a novel SAR image denoising algorithm based on an enhanced residual network architecture,with the objective of enhancing the utility of SAR imagery in complex electromagnetic environments.The proposed algorithm integrates residual network modules,which directly process the noisy input images to generate denoised outputs.This approach not only reduces computational complexity but also mitigates the difficulties associated with model training.By combining the Transformer module with the residual block,the algorithm enhances the network's ability to extract global features,offering superior feature extraction capabilities compared to CNN-based residual modules.Additionally,the algorithm employs the adaptive activation function Meta-ACON,which dynamically adjusts the activation patterns of neurons,thereby improving the network's feature extraction efficiency.The effectiveness of the proposed denoising method is empirically validated using real SAR images from the RSOD dataset.The proposed algorithm exhibits remarkable performance in terms of EPI,SSIM,and ENL,while achieving a substantial enhancement in PSNR when compared to traditional and deep learning-based algorithms.The PSNR performance is enhanced by over twofold.Moreover,the evaluation of the MSTAR SAR dataset substantiates the algorithm's robustness and applicability in SAR denoising tasks,with a PSNR of 25.2021 being attained.These findings underscore the efficacy of the proposed algorithm in mitigating speckle noise while preserving critical features in SAR imagery,thereby enhancing its quality and usability in practical scenarios.
基金This work was partially supported by the research grant of the National University of Singapore(NUS),Ministry of Education(MOE Tier 1).
文摘To improve the performance of multilayer perceptron(MLP)neural networks activated by conventional activation functions,this paper presents a new MLP activated by univariate Gaussian radial basis functions(RBFs)with adaptive centers and widths,which is composed of more than one hidden layer.In the hidden layer of the RBF-activated MLP network(MLPRBF),the outputs of the preceding layer are first linearly transformed and then fed into the univariate Gaussian RBF,which exploits the highly nonlinear property of RBF.Adaptive RBFs might address the issues of saturated outputs,low sensitivity,and vanishing gradients in MLPs activated by other prevailing nonlinear functions.Finally,we apply four MLP networks with the rectified linear unit(ReLU),sigmoid function(sigmoid),hyperbolic tangent function(tanh),and Gaussian RBF as the activation functions to approximate the one-dimensional(1D)sinusoidal function,the analytical solution of viscous Burgers’equation,and the two-dimensional(2D)steady lid-driven cavity flows.Using the same network structure,MLP-RBF generally predicts more accurately and converges faster than the other threeMLPs.MLP-RBF using less hidden layers and/or neurons per layer can yield comparable or even higher approximation accuracy than other MLPs equipped with more layers or neurons.
文摘Machine Learning(ML)and Deep Learning(DL)technologies are revolutionizing the medical domain,especially with Electrocardiogram(ECG),by providing new tools and techniques for diagnosing,treating,and preventing diseases.However,DL architectures are computationally more demanding.In recent years,researchers have focused on combining the computationally less intensive portion of the DL architectures with ML approaches,say for example,combining the convolutional layer blocks of Convolution Neural Networks(CNNs)into ML algorithms such as Extreme Gradient Boosting(XGBoost)and K-Nearest Neighbor(KNN)resulting in CNN-XGBoost and CNN-KNN,respectively.However,these approaches are homogenous in the sense that they use a fixed Activation Function(AFs)in the sequence of convolution and pooling layers,thereby limiting the ability to capture unique features.Since various AFs are readily available and each could capture unique features,we propose a Convolutionbased Heterogeneous Activation Facility(CHAF)which uses multiple AFs in the convolution layer blocks,one for each block,with a motivation of extracting features in a better manner to improve the accuracy.The proposed CHAF approach is validated on PTB and shown to outperform the homogeneous approaches such as CNN-KNN and CNN-XGBoost.For PTB dataset,proposed CHAF-KNN has an accuracy of 99.55%and an F1 score of 99.68%in just 0.008 s,outperforming the state-of-the-art CNN-XGBoost which has an accuracy of 99.38%and an F1 score of 99.32%in 1.23 s.To validate the generality of the proposed CHAF,experiments were repeated on MIT-BIH dataset,and the proposed CHAF-KNN is shown to outperform CNN-KNN and CNN-XGBoost.
基金School-level Educational Reform Project of Hangzhou Normal University(Project No.:HLXYJG202304)。
文摘Objective:To conduct a scoping review on the application status of the Functional Activity Score(FAS)in postoperative active pain management in China,providing a reference for its standardized and normative promotion.Methods:Computerized searches of Chinese and English databases were performed to collect studies published by Chinese scholars from 2005 to July 2025 on the application of FAS in postoperative active pain management.After strict screening,the basic characteristics,application fields,assessment models,evaluation timing,types of functional activities,and clinical outcomes of the included literature were systematically analyzed.Results:A total of 18 studies were included,involving surgical types such as thoracic surgery,general surgery,and orthopedics.All studies adopted FAS combined with the Numeric Rating Scale(NRS)for assessment,with evaluation timing mostly concentrated within 72 hours postoperatively.The selected functional activities primarily included respiration-related and limb movements.Evaluation indicators covered pain control,functional recovery,complications,adverse events,patient experience,and tool assessment,with most studies reporting positive outcomes.Conclusion:FAS can effectively enhance pain control and promote functional recovery in postoperative active pain management in China,demonstrating high clinical value.However,existing studies exhibit inconsistencies in assessment criteria,selection of activity types,and research quality.
