摘要
中文分词是中文信息处理的前提和基础.文章综合基于词典的分词方法和统计的分词方法的优点,旨在提出一种基于词性标注与分词消歧的中文分词方法,针对初次分词结果中的歧义集进行对比消歧.利用正向、逆向最大匹配方法以及隐马尔可夫模型进行分词,得到三种分词结果中的歧义集.基于词信息对歧义集进行初次消歧.通过隐马尔可夫模型标注词性,利用Viterbi算法对每种分词结果选取最优词性标注.并基于定义的最优切分评估函数对歧义集进行二次消歧,从而获得最终分词.通过实验对比,证明该方法具有良好的消歧效果.
Chinese words segmentation is the premise and basis of Chinese information processing.This paper puts forward a method of segmentation based on part of speech tagging and sense disambiguation for ambiguity sets,which combines the advantage of method based on the dictionary and method based on statistics for Chinese words segmentation.This method segments word based on FMM,RMM and HMM at first.Then it compares the segmentation results with each other,and output the optimal segmentation by word information and the definition of optimal estimation function of part of speech tagging.Experimental results show that this method is able to reduce the error of sense disambiguation.
作者
熊健
翟紫姹
XIONG Jian;ZHAI Zi-cha(School of Economics and Statistics,Guangzhou University,Guangzhou 510006,China)
出处
《广州大学学报（自然科学版）》
CAS
2019年第5期27-33,共7页
Journal of Guangzhou University:Natural Science Edition
关键词
分词
分词消歧
最大匹配法
隐马尔可夫模型
词性标注
word segmentation
sense disambiguation
maximum matching method
HMM
part of speech tagging
