针对低资源语言缺少标签数据,而无法使用现有成熟的深度学习方法进行命名实体识别(NER)的问题,提出基于句级别对抗生成网络(GAN)的跨语言NER模型——SLGAN-XLM-R(Sentence Level GAN Based on XLM-R)。首先,使用源语言的标签数据在预训...针对低资源语言缺少标签数据,而无法使用现有成熟的深度学习方法进行命名实体识别(NER)的问题,提出基于句级别对抗生成网络(GAN)的跨语言NER模型——SLGAN-XLM-R(Sentence Level GAN Based on XLM-R)。首先,使用源语言的标签数据在预训练模型XLM-R (XLM-Robustly optimized BERT pretraining approach)的基础上训练NER模型;同时,结合目标语言的无标签数据对XLM-R模型的嵌入层进行语言对抗训练;然后,使用NER模型来预测目标语言无标签数据的软标签;最后,混合源语言与目标语言的标签数据,以对模型进行二次微调来得到最终的NER模型。在CoNLL2002和CoNLL2003两个数据集的英语、德语、西班牙语、荷兰语四种语言上的实验结果表明,以英语作为源语言时,SLGAN-XLM-R模型在德语、西班牙语、荷兰语测试集上的F1值分别为72.70%、79.42%、80.03%,相较于直接在XLM-R模型上进行微调分别提升了5.38、5.38、3.05个百分点。展开更多
Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requir...Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.展开更多
文摘针对低资源语言缺少标签数据,而无法使用现有成熟的深度学习方法进行命名实体识别(NER)的问题,提出基于句级别对抗生成网络(GAN)的跨语言NER模型——SLGAN-XLM-R(Sentence Level GAN Based on XLM-R)。首先,使用源语言的标签数据在预训练模型XLM-R (XLM-Robustly optimized BERT pretraining approach)的基础上训练NER模型;同时,结合目标语言的无标签数据对XLM-R模型的嵌入层进行语言对抗训练;然后,使用NER模型来预测目标语言无标签数据的软标签;最后,混合源语言与目标语言的标签数据,以对模型进行二次微调来得到最终的NER模型。在CoNLL2002和CoNLL2003两个数据集的英语、德语、西班牙语、荷兰语四种语言上的实验结果表明,以英语作为源语言时,SLGAN-XLM-R模型在德语、西班牙语、荷兰语测试集上的F1值分别为72.70%、79.42%、80.03%,相较于直接在XLM-R模型上进行微调分别提升了5.38、5.38、3.05个百分点。
文摘Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.
