A Long Short-Term Memory(LSTM) Recurrent Neural Network(RNN) has driven tremendous improvements on an acoustic model based on Gaussian Mixture Model(GMM). However, these models based on a hybrid method require a force...A Long Short-Term Memory(LSTM) Recurrent Neural Network(RNN) has driven tremendous improvements on an acoustic model based on Gaussian Mixture Model(GMM). However, these models based on a hybrid method require a forced aligned Hidden Markov Model(HMM) state sequence obtained from the GMM-based acoustic model. Therefore, it requires a long computation time for training both the GMM-based acoustic model and a deep learning-based acoustic model. In order to solve this problem, an acoustic model using CTC algorithm is proposed. CTC algorithm does not require the GMM-based acoustic model because it does not use the forced aligned HMM state sequence. However, previous works on a LSTM RNN-based acoustic model using CTC used a small-scale training corpus. In this paper, the LSTM RNN-based acoustic model using CTC is trained on a large-scale training corpus and its performance is evaluated. The implemented acoustic model has a performance of 6.18% and 15.01% in terms of Word Error Rate(WER) for clean speech and noisy speech, respectively. This is similar to a performance of the acoustic model based on the hybrid method.展开更多
最近,基于自注意力的Transformer结构在不同领域的一系列任务上表现出非常好的性能。探索了基于Transformer编码器和LAS(listen,attend and spell)解码器的Transformer-LAS语音识别模型的效果,并针对Transformer不善于捕捉局部信息的问...最近,基于自注意力的Transformer结构在不同领域的一系列任务上表现出非常好的性能。探索了基于Transformer编码器和LAS(listen,attend and spell)解码器的Transformer-LAS语音识别模型的效果,并针对Transformer不善于捕捉局部信息的问题,使用Conformer代替Transformer,提出Conformer-LAS模型。由于Attention过于灵活的对齐方式,使得在嘈杂环境中的效果急剧下降,采用连接时序分类(connectionist temporal classification,CTC)辅助训练以加快收敛,并加入音素级别的中间CTC损失联合优化,提出了效果更好的Conformer-LAS-CTC语音识别模型。在开源中文普通话Aishell-1数据集上对提出来的模型进行验证,实验结果表明,Conformer-LAS-CTC相对于采用的基线BLSTM-LAS和Transformer-LAS模型在测试集上的字错率分别相对降低了22.58%和48.76%,模型最终字错误率为4.54%。展开更多
Traditional speech recognition model based on deep neural network(DNN)and hidden Markov model(HMM)is a complex and multi-module system.In other words,optimization goals may differ between modules in traditional model....Traditional speech recognition model based on deep neural network(DNN)and hidden Markov model(HMM)is a complex and multi-module system.In other words,optimization goals may differ between modules in traditional model.Besides,additional language resources are required,such as pronunciation dictionary and language model.To eliminate the drawbacks of traditional model,we hereby propose an end-to-end speech recognition method,where connectionist temporal classification(CTC)and attention are integrated for decoding.In our model,the complex modules are replaced by a single deep network.Our model mainly consists of encoder and decoder.The encoder is constructed by bidirectional long short-term memory(BLSTM)with a triangular structure for feature extraction.The decoder based on CTC-attention decoding utilizes advanced features extracted by shared encoder for training and decoding.The experimental results on the Vox Forge dataset indicate that end-to-end method is superior to basic CTC and attention-based encoder-decoder decoding,and the character error rate(CER)is reduced to 12.9%without using any language model.展开更多
Continuous sign language recognition(CSLR)is challenging due to the complexity of video background,hand gesture variability,and temporal modeling difficulties.This work proposes a CSLR method based on a spatialtempora...Continuous sign language recognition(CSLR)is challenging due to the complexity of video background,hand gesture variability,and temporal modeling difficulties.This work proposes a CSLR method based on a spatialtemporal graph attention network to focus on essential features of video series.The method considers local details of sign language movements by taking the information on joints and bones as inputs and constructing a spatialtemporal graph to reflect inter-frame relevance and physical connections between nodes.The graph-based multihead attention mechanism is utilized with adjacent matrix calculation for better local-feature exploration,and short-term motion correlation modeling is completed via a temporal convolutional network.We adopted BLSTM to learn the long-termdependence and connectionist temporal classification to align the word-level sequences.The proposed method achieves competitive results regarding word error rates(1.59%)on the Chinese Sign Language dataset and the mean Jaccard Index(65.78%)on the ChaLearn LAP Continuous Gesture Dataset.展开更多
预训练模型通过自监督学习表示在非平行语料语音转换(VC)取得了重大突破。随着自监督预训练表示(SSPR)的广泛使用,预训练模型提取的特征中被证实包含更多的内容信息。提出一种基于SSPR同时结合矢量量化(VQ)和联结时序分类(CTC)的VC模型...预训练模型通过自监督学习表示在非平行语料语音转换(VC)取得了重大突破。随着自监督预训练表示(SSPR)的广泛使用,预训练模型提取的特征中被证实包含更多的内容信息。提出一种基于SSPR同时结合矢量量化(VQ)和联结时序分类(CTC)的VC模型。将预训练模型提取的SSPR作为端到端模型的输入,用于提高单次语音转换质量。如何有效地解耦内容表示和说话人表示成为语音转换中的关键问题。使用SSPR作为初步的内容信息,采用VQ从语音中解耦内容和说话人表示。然而,仅使用VQ只能将内容信息离散化,很难将纯粹的内容表示从语音中分离出来,为了进一步消除内容信息中说话人的不变信息,提出CTC损失指导内容编码器。CTC不仅作为辅助网络加快模型收敛,同时其额外的文本监督可以与VQ联合优化,实现性能互补,学习纯内容表示。说话人表示采用风格嵌入学习,2种表示作为系统的输入进行语音转换。在开源的CMU数据集和VCTK语料库对所提的方法进行评估,实验结果表明,该方法在客观上的梅尔倒谱失真(MCD)达到8.896 d B,在主观上的语音自然度平均意见分数(MOS)和说话人相似度MOS分别为3.29和3.22,均优于基线模型,此方法在语音转换的质量和说话人相似度上能够获得最佳性能。展开更多
基金supported by the Ministry of Trade,Industry & Energy(MOTIE,Korea) under Industrial Technology Innovation Program (No.10063424,'development of distant speech recognition and multi-task dialog processing technologies for in-door conversational robots')
文摘A Long Short-Term Memory(LSTM) Recurrent Neural Network(RNN) has driven tremendous improvements on an acoustic model based on Gaussian Mixture Model(GMM). However, these models based on a hybrid method require a forced aligned Hidden Markov Model(HMM) state sequence obtained from the GMM-based acoustic model. Therefore, it requires a long computation time for training both the GMM-based acoustic model and a deep learning-based acoustic model. In order to solve this problem, an acoustic model using CTC algorithm is proposed. CTC algorithm does not require the GMM-based acoustic model because it does not use the forced aligned HMM state sequence. However, previous works on a LSTM RNN-based acoustic model using CTC used a small-scale training corpus. In this paper, the LSTM RNN-based acoustic model using CTC is trained on a large-scale training corpus and its performance is evaluated. The implemented acoustic model has a performance of 6.18% and 15.01% in terms of Word Error Rate(WER) for clean speech and noisy speech, respectively. This is similar to a performance of the acoustic model based on the hybrid method.
文摘最近,基于自注意力的Transformer结构在不同领域的一系列任务上表现出非常好的性能。探索了基于Transformer编码器和LAS(listen,attend and spell)解码器的Transformer-LAS语音识别模型的效果,并针对Transformer不善于捕捉局部信息的问题,使用Conformer代替Transformer,提出Conformer-LAS模型。由于Attention过于灵活的对齐方式,使得在嘈杂环境中的效果急剧下降,采用连接时序分类(connectionist temporal classification,CTC)辅助训练以加快收敛,并加入音素级别的中间CTC损失联合优化,提出了效果更好的Conformer-LAS-CTC语音识别模型。在开源中文普通话Aishell-1数据集上对提出来的模型进行验证,实验结果表明,Conformer-LAS-CTC相对于采用的基线BLSTM-LAS和Transformer-LAS模型在测试集上的字错率分别相对降低了22.58%和48.76%,模型最终字错误率为4.54%。
文摘Traditional speech recognition model based on deep neural network(DNN)and hidden Markov model(HMM)is a complex and multi-module system.In other words,optimization goals may differ between modules in traditional model.Besides,additional language resources are required,such as pronunciation dictionary and language model.To eliminate the drawbacks of traditional model,we hereby propose an end-to-end speech recognition method,where connectionist temporal classification(CTC)and attention are integrated for decoding.In our model,the complex modules are replaced by a single deep network.Our model mainly consists of encoder and decoder.The encoder is constructed by bidirectional long short-term memory(BLSTM)with a triangular structure for feature extraction.The decoder based on CTC-attention decoding utilizes advanced features extracted by shared encoder for training and decoding.The experimental results on the Vox Forge dataset indicate that end-to-end method is superior to basic CTC and attention-based encoder-decoder decoding,and the character error rate(CER)is reduced to 12.9%without using any language model.
基金supported by the Key Research&Development Plan Project of Shandong Province,China(No.2017GGX10127).
文摘Continuous sign language recognition(CSLR)is challenging due to the complexity of video background,hand gesture variability,and temporal modeling difficulties.This work proposes a CSLR method based on a spatialtemporal graph attention network to focus on essential features of video series.The method considers local details of sign language movements by taking the information on joints and bones as inputs and constructing a spatialtemporal graph to reflect inter-frame relevance and physical connections between nodes.The graph-based multihead attention mechanism is utilized with adjacent matrix calculation for better local-feature exploration,and short-term motion correlation modeling is completed via a temporal convolutional network.We adopted BLSTM to learn the long-termdependence and connectionist temporal classification to align the word-level sequences.The proposed method achieves competitive results regarding word error rates(1.59%)on the Chinese Sign Language dataset and the mean Jaccard Index(65.78%)on the ChaLearn LAP Continuous Gesture Dataset.
文摘预训练模型通过自监督学习表示在非平行语料语音转换(VC)取得了重大突破。随着自监督预训练表示(SSPR)的广泛使用,预训练模型提取的特征中被证实包含更多的内容信息。提出一种基于SSPR同时结合矢量量化(VQ)和联结时序分类(CTC)的VC模型。将预训练模型提取的SSPR作为端到端模型的输入,用于提高单次语音转换质量。如何有效地解耦内容表示和说话人表示成为语音转换中的关键问题。使用SSPR作为初步的内容信息,采用VQ从语音中解耦内容和说话人表示。然而,仅使用VQ只能将内容信息离散化,很难将纯粹的内容表示从语音中分离出来,为了进一步消除内容信息中说话人的不变信息,提出CTC损失指导内容编码器。CTC不仅作为辅助网络加快模型收敛,同时其额外的文本监督可以与VQ联合优化,实现性能互补,学习纯内容表示。说话人表示采用风格嵌入学习,2种表示作为系统的输入进行语音转换。在开源的CMU数据集和VCTK语料库对所提的方法进行评估,实验结果表明,该方法在客观上的梅尔倒谱失真(MCD)达到8.896 d B,在主观上的语音自然度平均意见分数(MOS)和说话人相似度MOS分别为3.29和3.22,均优于基线模型,此方法在语音转换的质量和说话人相似度上能够获得最佳性能。
