With the flooding of pornographic information on the Internet, how to keep people away from that offensive information is becoming one of the most important research areas in network information security. Some applica...With the flooding of pornographic information on the Internet, how to keep people away from that offensive information is becoming one of the most important research areas in network information security. Some applications which can block or filter such information are used. Approaches in those systems can be roughly classified into two kinds: metadata based and content based. With the development of distributed technologies, content based filtering technologies will play a more and more important role in filtering systems. Keyword matching is a content based method used widely in harmful text filtering. Experiments to evaluate the recall and precision of the method showed that the precision of the method is not satisfactory, though the recall of the method is rather high. According to the results, a new pornographic text filtering model based on reconfirming is put forward. Experiments showed that the model is practical, has less loss of recall than the single keyword matching method, and has higher precision.展开更多
With the development of Internet technology,the explosive growth of Internet information presentation has led to difficulty in filtering effective information.Finding a model with high accuracy for text classification...With the development of Internet technology,the explosive growth of Internet information presentation has led to difficulty in filtering effective information.Finding a model with high accuracy for text classification has become a critical problem to be solved by text filtering,especially for Chinese texts.This paper selected the manually calibrated Douban movie website comment data for research.First,a text filtering model based on the BP neural network has been built;Second,based on the Term Frequency-Inverse Document Frequency(TF-IDF)vector space model and the doc2vec method,the text word frequency vector and the text semantic vector were obtained respectively,and the text word frequency vector was linearly reduced by the Principal Component Analysis(PCA)method.Third,the text word frequency vector after dimensionality reduction and the text semantic vector were combined,add the text value degree,and the text synthesis vector was constructed.Experiments show that the model combined with text word frequency vector degree after dimensionality reduction,text semantic vector,and text value has reached the highest accuracy of 84.67%.展开更多
基于检索增强生成(RAG)的军事领域知识问答系统已经逐渐成为现代情报人员收集和分析情报的重要工具。针对目前RAG方法的应用策略中的混合检索存在可移植性不强以及非必要使用查询改写容易诱发语义漂移的问题,提出一种多策略检索增强生成...基于检索增强生成(RAG)的军事领域知识问答系统已经逐渐成为现代情报人员收集和分析情报的重要工具。针对目前RAG方法的应用策略中的混合检索存在可移植性不强以及非必要使用查询改写容易诱发语义漂移的问题,提出一种多策略检索增强生成(MSRAG)方法。首先,根据用户输入的查询特点自适应地匹配检索模型来召回相关文本;其次,利用文本过滤器提取出能够回答问题的关键文本片段;再次,使用文本过滤器进行内容有效性判断以启动基于同义词拓展的查询改写,并将初始查询与改写后的信息合并输入检索控制器以进行更有针对性的再次检索;最后,合并能够回答问题的关键文本片段和问题,并使用提示工程输入生成答案模型来生成响应返回给用户。实验结果表明,MSRAG方法在军事领域数据集(Military)和Medical数据集的ROUGE-L(Recall-Oriented Understudy for Gisting Evaluation Longest common subsequence)指标上相较于凸线性组合RAG方法分别提高了14.35和5.83个百分点。可见,MSRAG方法具备较强的通用性和可移植性,能够缓解非必要查询改写导致的语义漂移现象,有效帮助大模型生成更准确的答案。展开更多
针对文档图像文本行分割方法在处理书写方式多样的多语种古籍时效果不佳的问题,提出一种基于全局滤波器和视觉置换器的多语种古籍文档图像的文本行切分方法(Global Filter and Vision Permutator based U-Net,GVUnet)。构建一个多语种...针对文档图像文本行分割方法在处理书写方式多样的多语种古籍时效果不佳的问题,提出一种基于全局滤波器和视觉置换器的多语种古籍文档图像的文本行切分方法(Global Filter and Vision Permutator based U-Net,GVUnet)。构建一个多语种古籍文档图像文本行分割数据集,包含汉字、古布依文、满文这3个类别,每个类别各50张图像,共150张。GVUnet利用全局滤波器和视觉置换器在提取全局信息的同时,关注文本行位置信息,对古籍文档进行有效分割。实验结果表明,该文提出的文本行分割方法在新构建的数据集上mIoU(mean intersection over union,mIoU)得分达到92.78%,可以有效处理排版方式不同的汉字、古布依文、满文古籍文档。展开更多
文摘With the flooding of pornographic information on the Internet, how to keep people away from that offensive information is becoming one of the most important research areas in network information security. Some applications which can block or filter such information are used. Approaches in those systems can be roughly classified into two kinds: metadata based and content based. With the development of distributed technologies, content based filtering technologies will play a more and more important role in filtering systems. Keyword matching is a content based method used widely in harmful text filtering. Experiments to evaluate the recall and precision of the method showed that the precision of the method is not satisfactory, though the recall of the method is rather high. According to the results, a new pornographic text filtering model based on reconfirming is put forward. Experiments showed that the model is practical, has less loss of recall than the single keyword matching method, and has higher precision.
基金Supported by the Sichuan Science and Technology Program (2021YFQ0003).
文摘With the development of Internet technology,the explosive growth of Internet information presentation has led to difficulty in filtering effective information.Finding a model with high accuracy for text classification has become a critical problem to be solved by text filtering,especially for Chinese texts.This paper selected the manually calibrated Douban movie website comment data for research.First,a text filtering model based on the BP neural network has been built;Second,based on the Term Frequency-Inverse Document Frequency(TF-IDF)vector space model and the doc2vec method,the text word frequency vector and the text semantic vector were obtained respectively,and the text word frequency vector was linearly reduced by the Principal Component Analysis(PCA)method.Third,the text word frequency vector after dimensionality reduction and the text semantic vector were combined,add the text value degree,and the text synthesis vector was constructed.Experiments show that the model combined with text word frequency vector degree after dimensionality reduction,text semantic vector,and text value has reached the highest accuracy of 84.67%.
文摘基于检索增强生成(RAG)的军事领域知识问答系统已经逐渐成为现代情报人员收集和分析情报的重要工具。针对目前RAG方法的应用策略中的混合检索存在可移植性不强以及非必要使用查询改写容易诱发语义漂移的问题,提出一种多策略检索增强生成(MSRAG)方法。首先,根据用户输入的查询特点自适应地匹配检索模型来召回相关文本;其次,利用文本过滤器提取出能够回答问题的关键文本片段;再次,使用文本过滤器进行内容有效性判断以启动基于同义词拓展的查询改写,并将初始查询与改写后的信息合并输入检索控制器以进行更有针对性的再次检索;最后,合并能够回答问题的关键文本片段和问题,并使用提示工程输入生成答案模型来生成响应返回给用户。实验结果表明,MSRAG方法在军事领域数据集(Military)和Medical数据集的ROUGE-L(Recall-Oriented Understudy for Gisting Evaluation Longest common subsequence)指标上相较于凸线性组合RAG方法分别提高了14.35和5.83个百分点。可见,MSRAG方法具备较强的通用性和可移植性,能够缓解非必要查询改写导致的语义漂移现象,有效帮助大模型生成更准确的答案。
文摘针对文档图像文本行分割方法在处理书写方式多样的多语种古籍时效果不佳的问题,提出一种基于全局滤波器和视觉置换器的多语种古籍文档图像的文本行切分方法(Global Filter and Vision Permutator based U-Net,GVUnet)。构建一个多语种古籍文档图像文本行分割数据集,包含汉字、古布依文、满文这3个类别,每个类别各50张图像,共150张。GVUnet利用全局滤波器和视觉置换器在提取全局信息的同时,关注文本行位置信息,对古籍文档进行有效分割。实验结果表明,该文提出的文本行分割方法在新构建的数据集上mIoU(mean intersection over union,mIoU)得分达到92.78%,可以有效处理排版方式不同的汉字、古布依文、满文古籍文档。
