摘要
针对深度学习应用技术进行了研究性综述。详细阐述了RBM(受限玻尔兹曼机)逐层预训练后再用BP(反向传播)微调的深度学习贪婪层训练方法,对比分析了BP算法中三种梯度下降的方式,建议在线学习系统采用随机梯度下降,静态离线学习系统采用随机小批量梯度下降;归纳总结了深度学习深层结构特征,并推荐了目前最受欢迎的五层深度网络结构设计方法。分析了前馈神经网络非线性激活函数的必要性及常用的激活函数优点,并推荐Re LU(rectified linear units)激活函数。最后简要概括了深度卷积神经网络、深度递归神经网络、长短期记忆网络等新型深度网络的特点及应用场景,归纳总结了当前深度学习可能的发展方向。
This paper reviewed the deep learning algorithms and their applications. It elaborated the greedy layer training al- gorithm which used the fine-grained back-propagation （BP） learning following the layer-wise pre-training on each restricted Bohzmann machine （RBM） layer. After comparing and analyzing the three ways of gradient descent in the BP algorithm, this paper suggested applying stochastic gradient descent in online learning and adopting stochastic mini-batch gradient descent in static offline learning. It summarized the characteristic of the network structure in deep learning and recommend the design of state-of-art five-layer network architecture. It also analyzed the necessity of the nonlinear activation function in feedforward neural networks and the advantages of the common activation functions, and recommended using ReLU activate function. Fi- nally, the paper provided a brief summary of features and application scenarios of emerging deep neural networks such as deep CNN （convolutional neural networks） , deep RNNs（ recurrent neural networks） and LSTM （long short-termmemory networks） , as well as the potential directions of future deep learning applications and research.
出处
《计算机应用研究》
CSCD
北大核心
2016年第11期3201-3205,共5页
Application Research of Computers
基金
国家自然科学基金资助项目(61472316
61172090)
国家科技重大专项基金资助项目(2012ZX03002001)
高等教育博士点研究基金资助项目(20120201110013)
陕西省自然科学基金资助项目(2014JM1006
2014KRM28-01)
中央高校基本科研业务费专项资金资助项目(XKJC2014008)
陕西省自然科学创新工程资助项目(2013SZS16-Z01/P01/K01)
