Chinese text categorization differs from English text categorization due to its much larger term set （of words or character n-grams）, which results in very slow training and working of modern high-performance classi...Chinese text categorization differs from English text categorization due to its much larger term set （of words or character n-grams）, which results in very slow training and working of modern high-performance classifiers. This study assumes that this high-dimensionality problem is related to the redundancy in the term set, which cannot be solved by traditional term selection methods. A greedy algorithm framework named ＂non-independent term selection＂ is presented, which reduces the redundancy according to string-level correlations. Several preliminary implementations of this idea are demonstrated. Experiment results show that a good tradeoff can be reached between the performance and the size of the term set.展开更多
基金Supported by the National Natural Science Foundation of China(Nos. 60573187 and 60321002)the National High-Tech Research and Development (863) Program of China (No.2007AA01Z148)
文摘Chinese text categorization differs from English text categorization due to its much larger term set （of words or character n-grams）, which results in very slow training and working of modern high-performance classifiers. This study assumes that this high-dimensionality problem is related to the redundancy in the term set, which cannot be solved by traditional term selection methods. A greedy algorithm framework named ＂non-independent term selection＂ is presented, which reduces the redundancy according to string-level correlations. Several preliminary implementations of this idea are demonstrated. Experiment results show that a good tradeoff can be reached between the performance and the size of the term set.
