摘要
提出基于同源网页去重与内容去重的策略.通过对网址URL进行哈希散列完成对同源网页的去重,并对内容相同或近似的网页采用基于主题概念的去重判断.实验表明,该方法有效且去重效果良好.基于上述算法实现了教育资源库教育资讯搜索引擎系统.
This paper presented techniques on how to build an effective crawler to collect non-replicative Web pages. A novel Hash function was proposed, together with a content-oriented approach, to filter based on URLs and contents. On one hand, this technique can parallelize crawling process while minimize the overlap effectively. On the other hand, it can identify those near-duplicated collections. The experimental results show the feasibility of the approach. On the basis of this work, the implementation of an educational search engine was presented in the end.
出处
《上海交通大学学报》
EI
CAS
CSCD
北大核心
2006年第5期775-777,782,共4页
Journal of Shanghai Jiaotong University
基金
国家高技术研究发展计划(863)项目(2002AA119050)
关键词
信息检索
搜索引擎
哈希函数
主题概念
information retrieval
search engine
Hash function
subject concept
