摘要
首先对网页源码文本统一编码转为UTF格式,然后把HTML网页文档转换为XML文档并解析为一棵DOM树。依据XML语言特点及噪声特征规则先对DOM树的噪声节点进行过滤删除,然后依据中文标点符号统计方法提取网页正文内容,并在此基础上利用行文本统计方法去除提取出的正文中存在的噪声信息,最后得到网页正文文本。对来自结构完全不同的主流与非主流的中英文新闻网站上的2 000篇网页进行实验,结果表明本文提出的方法具有较高的抽取准确率,并具有很好的通用性和实现简单的特点,适用于针对互联网中不同网站新闻文本信息的自动采集。
As different web pages have different codes, the HTML web page first need to be encoded with the uniform code UTFS, and then translated into an XML document which is parsed into the DOM tree. After removing some noise nodes from the DOM tree according to the features of XML language and the rules of the noise characteristics, text con- tents are extracted from the DOM tree by the method of statistics of punctuation and noise information is continued to be eliminated from contents extracted above by the method of statistics of line-text. The result of experiments on 2000 web pages obtained from different web sites shows that our method has high accuracy, great generality, and simplicity, and can be automatically used to extract the right contents from different web sites.
出处
《山东大学学报（理学版）》
CAS
CSCD
北大核心
2012年第3期38-42,共5页
Journal of Shandong University(Natural Science)
基金
国家自然科学基金资助项目(61070061)
教育部人文社会科学研究青年基金资助项目(11YJCZH086)
广州社科青年基金资助项目(11Q20)
关键词
网页文本抽取
DOM树
行文本统计
标点符号统计
content extraction from web pages
DOM tree
statistical of line-text
statistical of punctuation
