Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm...Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm based on Spark big data platform.Since the TF-IDF(term frequency-inverse document frequency)algorithm under Spark is irreversible to word mapping,the mapped words indexes cannot be traced back to the original words.In this paper,an optimized method is proposed that TF-IDF under Spark to ensure the text words can be restored.Firstly,the text feature is extracted by the TF-IDF algorithm combined CountVectorizer proposed in this paper,and then the features are inputted to the LDA(Latent Dirichlet Allocation)topic model for training.Finally,the text topic clustering is obtained.Experimental results show that for large data samples,the processing speed of LDA topic model clustering has been improved based Spark.At the same time,compared with the LDA topic model based on word frequency input,the model proposed in this paper has a reduction of perplexity.展开更多
Single-pass is commonly used in topic detection and tracking( TDT) due to its simplicity,high efficiency and low cost. When dealing with large-scale data,time cost will increase sharply and clustering performance will...Single-pass is commonly used in topic detection and tracking( TDT) due to its simplicity,high efficiency and low cost. When dealing with large-scale data,time cost will increase sharply and clustering performance will be affected greatly. Aiming at this problem,hierarchical clustering algorithm based on single-pass is proposed,which is inspired by hierarchical and concurrent ideas to divide clustering process into three stages. News reports are classified into different categories firstly.Then there are twice single-pass clustering processes in the same category,and one agglomerative clustering among different categories. In addition,for semantic similarity in news reports,topic model is improved based on named entities. Experimental results show that the proposed method can effectively accelerate the process as well as improve the performance.展开更多
Modeling topics in short texts presents significant challenges due to feature sparsity, particularly when analyzing content generated by large-scale online users. This sparsity can substantially impair semantic captur...Modeling topics in short texts presents significant challenges due to feature sparsity, particularly when analyzing content generated by large-scale online users. This sparsity can substantially impair semantic capture accuracy. We propose a novel approach that incorporates pre-clustered knowledge into the BERTopic model while reducing the l2 norm for low-frequency words. Our method effectively mitigates feature sparsity during cluster mapping. Empirical evaluation on the StackOverflow dataset demonstrates that our approach outperforms baseline models, achieving superior Macro-F1 scores. These results validate the effectiveness of our proposed feature sparsity reduction technique for short-text topic modeling.展开更多
针对现有的深度文档聚类方法在处理动态文档数据时,文档主题随时间演化过程中存在主题混淆和对齐不准确问题,提出一种深度演化主题聚类模型(DETCM)。DETCM可以捕捉动态文档随时间演化的主题信息,结合历史主题信息与当前时间片文档特征,...针对现有的深度文档聚类方法在处理动态文档数据时,文档主题随时间演化过程中存在主题混淆和对齐不准确问题,提出一种深度演化主题聚类模型(DETCM)。DETCM可以捕捉动态文档随时间演化的主题信息,结合历史主题信息与当前时间片文档特征,发掘事件主题演化脉络,生成动态文档主题表示。具体来说,为了解决主题随时间演变时的主题混淆问题,设计了基于混合编码器的主题融合学习模块,借助前置时间片的主题信息,加强当前时间片的主题区分度与特征提取。此外,还设计了一种跨时间片的动态主题继承模块,通过将不同时间片上的主题匹配对齐,有效地将历史时间片上的主题信息融入当前时间片的类簇划分过程中。这一设计使得DETCM学习主题时能够继承并借鉴历史时间片的主题信息,有效跟踪动态文本主题持续演化的过程。基于arXiv真实演化文本文档数据集的实验结果表明,相较于深度演化聚类模型DEDC-IMAE(Deep Evolutionary Document Clustering model with Instance-level Mutual Attention Enhancement),DETCM在所有时间片上的标准化互信息(NMI)指标平均提升了3.08%(-0.37%~5.43%),验证了DETCM在动态场景中具有更好的主题演化追踪能力,能够更准确地捕捉主题的时序变化特征,实现更优的聚类性能。展开更多
Purpose: Formal concept analysis(FCA) and concept lattice theory(CLT) are introduced for constructing a network of IDR topics and for evaluating their effectiveness for knowledge structure exploration.Design/methodolo...Purpose: Formal concept analysis(FCA) and concept lattice theory(CLT) are introduced for constructing a network of IDR topics and for evaluating their effectiveness for knowledge structure exploration.Design/methodology/approach: We introduced the theory and applications of FCA and CLT, and then proposed a method for interdisciplinary knowledge discovery based on CLT. As an example of empirical analysis, interdisciplinary research(IDR) topics in Information & Library Science(LIS) and Medical Informatics, and in LIS and Geography-Physical, were utilized as empirical fields. Subsequently, we carried out a comparative analysis with two other IDR topic recognition methods.Findings: The CLT approach is suitable for IDR topic identification and predictions.Research limitations: IDR topic recognition based on the CLT is not sensitive to the interdisciplinarity of topic terms, since the data can only reflect whether there is a relationship between the discipline and the topic terms. Moreover, the CLT cannot clearly represent a large amounts of concepts.Practical implications: A deeper understanding of the IDR topics was obtained as the structural and hierarchical relationships between them were identified, which can help to get more precise identification and prediction to IDR topics.Originality/value: IDR topics identification based on CLT have performed well and this theory has several advantages for identifying and predicting IDR topics. First, in a concept lattice, there is a partial order relation between interconnected nodes, and consequently, a complete concept lattice can present hierarchical properties. Second, clustering analysis of IDR topics based on concept lattices can yield clusters that highlight the essential knowledge features and help display the semantic relationship between different IDR topics. Furthermore, the Hasse diagram automatically displays all the IDR topics associated with the different disciplines, thus forming clusters of specific concepts and visually retaining and presenting the associations of IDR topics through multiple inheritance relationships between the concepts.展开更多
高价值专利识别是抢占产业全球战略高地、推动产业持续高效健康发展的重要课题,可为产业关键核心技术的挖掘提供重要线索。本文从专利技术距离测度视角出发,在进行主题聚类提取领域上位类主题基础上,提出一种基于主题知识贡献距离与主...高价值专利识别是抢占产业全球战略高地、推动产业持续高效健康发展的重要课题,可为产业关键核心技术的挖掘提供重要线索。本文从专利技术距离测度视角出发,在进行主题聚类提取领域上位类主题基础上,提出一种基于主题知识贡献距离与主题联系程度双维影响下的高价值专利识别方法。在主题知识贡献距离维度上,构建专利间分层专利引用网络,计算各专利与主题的持续知识贡献值,基于知识贡献时间序列计算主题间的动态时间规整(dynamic time warping,DTW)距离,形成主题知识贡献距离矩阵;在主题联系程度维度上,构建主题与专利二分图网络,结合专利共现频率与引用关系强度进行初始强度与全局逻辑计算,形成主题联系程度矩阵。融合双维度矩阵构建专利技术距离矩阵,基于技术距离矩阵进行各专利的绝对技术距离计算,选取阈值范围内的高绝对技术距离专利作为领域内高技术价值的专利。经验证数据集检验,本文方法的精准率达到0.8218,F1指标达到0.8014。基于此,对“生成式人工智能”领域专利进行实证,识别出产业内具有较高价值的专利1437件,并发现识别出的高价值专利集具有较高的转让比例,转让比例达58.59%。本文基于技术本质的视角对专利间的技术差距进行量化,打破了以往仅从外部特征或简单统计数据判断专利价值的局限性,提升了识别的准确性;同时,提出双维度的技术距离影响机理,进一步提升了识别的可解释性。展开更多
微信群组中存在大量会话文本数据,对其进行关键词提取有助于理解群组动态和主题演变。由于微信会话文本存在长度短、主题交叉、语言不规范等特点,传统提取方法效果欠佳。为此,提出了一个基于会话主题聚类的多阶段关键词提取算法。首先,...微信群组中存在大量会话文本数据,对其进行关键词提取有助于理解群组动态和主题演变。由于微信会话文本存在长度短、主题交叉、语言不规范等特点,传统提取方法效果欠佳。为此,提出了一个基于会话主题聚类的多阶段关键词提取算法。首先,提出了一种结合预训练知识的会话主题聚类算法(Single Pass Using Thread Segmentation and Pre-training Knowledge,SP_(TSPK)),综合考虑语义相关性、消息活跃度和用户亲密度,有效解决了会话主题交叉和信息量不足的问题。其次,设计了一种多阶段关键词提取算法(Multi-Stage Keyword Extraction,MSKE),将任务分解为无监督关键词抽取和有监督关键词生成,有效提取原文中存在和缺失的关键词,减少了候选词规模和语义冗余;最终,组合SP_(TSPK)算法与MSKE算法实现微信会话文本关键词提取。在WeChat数据集上相比AutoKeyGen算法,F_(1)@5和F_(1)@O平均提升了12.8%与10.8%,R@10平均达到其2.59倍。实验结果表明,该算法能有效地提取微信会话文本关键词。展开更多
【背景】蛹虫草(Cordyceps militaris)作为虫草科虫草属的模式种一直受到全球研究人员的关注。【目的】多维度探讨蛹虫草研究的当前状况与未来趋势。【方法】基于Web of Science核心合集数据库对2005-2024年间有关蛹虫草的SCI核心集论...【背景】蛹虫草(Cordyceps militaris)作为虫草科虫草属的模式种一直受到全球研究人员的关注。【目的】多维度探讨蛹虫草研究的当前状况与未来趋势。【方法】基于Web of Science核心合集数据库对2005-2024年间有关蛹虫草的SCI核心集论文进行了全面的数据搜集、整理、分析和可视化处理。【结果】过去20年里,蛹虫草研究已从单一的培养特性拓展至跨学科领域,尤其是其活性成分和药理学效应已成为学术界关注的焦点。文献计量分析结果显示,2005-2009年间,主要研究方向为蛹虫草的人工培养。2010-2014年间,研究主题扩展至子实体相关的药理学,研究地位显著提升。2015年后,研究主题进一步多元化,涵盖了优化、表达、氧化应激、真菌、抗氧化剂、化学成分、NF-κB、细胞周期停滞等领域,显示了从培养技术向深入的生物学和医学机制研究的转变。【结论】蛹虫草的研究经历了从传统培养研究向多学科交叉的深刻变革,未来研究将更加侧重于活性成分的功能机制、生物活性物质的药理作用及潜在的医学应用,为蛹虫草的深入研究和开发利用提供科学依据。展开更多
【背景】微生物浸出是处理尾矿、废弃矿石、低品位矿、难处理矿的有效浸出手段;相较于传统的浸出技术,其具有环境友好、收益好、冶金效率高等优点。【目的】深入探讨微生物浸出在全球范围内的发展趋势和学术影响,并协助研究人员确定研...【背景】微生物浸出是处理尾矿、废弃矿石、低品位矿、难处理矿的有效浸出手段;相较于传统的浸出技术,其具有环境友好、收益好、冶金效率高等优点。【目的】深入探讨微生物浸出在全球范围内的发展趋势和学术影响,并协助研究人员确定研究方向,开展相关研究,了解该领域最相关的课题。【方法】基于Web of Science核心合集数据库对2011-2023年全球微生物浸出的文献进行检索和分析。【结果】年发文趋势揭示微生物浸出领域的研究热度有所下降。高被引文献的研究表明,除了文献本身的质量外,另一个重要因素是国家政策支持和资金的可用性。全球共有80个国家包括1546个机构开展研究,共发表在580种期刊上。中国、伊朗、印度和澳大利亚进行了大量的研究,集成了冶金工程、环境科学生态学、采矿工程和生物技术与应用微生物学等多学科。聚类分析确定了4个经常出现的关键词:黄铜矿、废旧电路板、重金属和浸出,这为研究人员提供了新的检索词。【结论】目前对微生物浸出的研究主要集中在单个菌株,而混合菌株与矿物的吸附和耐受机理是未来发展所向。展开更多
基金This work is supported by the Science Research Projects of Hunan Provincial Education Department(Nos.18A174,18C0262)the National Natural Science Foundation of China(No.61772561)+2 种基金the Key Research&Development Plan of Hunan Province(Nos.2018NK2012,2019SK2022)the Degree&Postgraduate Education Reform Project of Hunan Province(No.209)the Postgraduate Education and Teaching Reform Project of Central South Forestry University(No.2019JG013).
文摘Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data,this paper takes news text as the research object and proposes LDA text topic clustering algorithm based on Spark big data platform.Since the TF-IDF(term frequency-inverse document frequency)algorithm under Spark is irreversible to word mapping,the mapped words indexes cannot be traced back to the original words.In this paper,an optimized method is proposed that TF-IDF under Spark to ensure the text words can be restored.Firstly,the text feature is extracted by the TF-IDF algorithm combined CountVectorizer proposed in this paper,and then the features are inputted to the LDA(Latent Dirichlet Allocation)topic model for training.Finally,the text topic clustering is obtained.Experimental results show that for large data samples,the processing speed of LDA topic model clustering has been improved based Spark.At the same time,compared with the LDA topic model based on word frequency input,the model proposed in this paper has a reduction of perplexity.
基金Supported by the National Natural Science Foundation of China(No.61502312)the Fundamental Research Funds for the Central Universities(No.2017BQ024)+1 种基金the Natural Science Foundation of Guangdong Province(No.2017A030310428)the Science and Technology Programm of Guangzhou(No.201806020075,20180210025)
文摘Single-pass is commonly used in topic detection and tracking( TDT) due to its simplicity,high efficiency and low cost. When dealing with large-scale data,time cost will increase sharply and clustering performance will be affected greatly. Aiming at this problem,hierarchical clustering algorithm based on single-pass is proposed,which is inspired by hierarchical and concurrent ideas to divide clustering process into three stages. News reports are classified into different categories firstly.Then there are twice single-pass clustering processes in the same category,and one agglomerative clustering among different categories. In addition,for semantic similarity in news reports,topic model is improved based on named entities. Experimental results show that the proposed method can effectively accelerate the process as well as improve the performance.
文摘Modeling topics in short texts presents significant challenges due to feature sparsity, particularly when analyzing content generated by large-scale online users. This sparsity can substantially impair semantic capture accuracy. We propose a novel approach that incorporates pre-clustered knowledge into the BERTopic model while reducing the l2 norm for low-frequency words. Our method effectively mitigates feature sparsity during cluster mapping. Empirical evaluation on the StackOverflow dataset demonstrates that our approach outperforms baseline models, achieving superior Macro-F1 scores. These results validate the effectiveness of our proposed feature sparsity reduction technique for short-text topic modeling.
文摘针对现有的深度文档聚类方法在处理动态文档数据时,文档主题随时间演化过程中存在主题混淆和对齐不准确问题,提出一种深度演化主题聚类模型(DETCM)。DETCM可以捕捉动态文档随时间演化的主题信息,结合历史主题信息与当前时间片文档特征,发掘事件主题演化脉络,生成动态文档主题表示。具体来说,为了解决主题随时间演变时的主题混淆问题,设计了基于混合编码器的主题融合学习模块,借助前置时间片的主题信息,加强当前时间片的主题区分度与特征提取。此外,还设计了一种跨时间片的动态主题继承模块,通过将不同时间片上的主题匹配对齐,有效地将历史时间片上的主题信息融入当前时间片的类簇划分过程中。这一设计使得DETCM学习主题时能够继承并借鉴历史时间片的主题信息,有效跟踪动态文本主题持续演化的过程。基于arXiv真实演化文本文档数据集的实验结果表明,相较于深度演化聚类模型DEDC-IMAE(Deep Evolutionary Document Clustering model with Instance-level Mutual Attention Enhancement),DETCM在所有时间片上的标准化互信息(NMI)指标平均提升了3.08%(-0.37%~5.43%),验证了DETCM在动态场景中具有更好的主题演化追踪能力,能够更准确地捕捉主题的时序变化特征,实现更优的聚类性能。
基金an outcome of the project "Study on the Recognition Method of Innovative Evolving Trajectory based on Topic Correlation Analysis of Science and Technology" (No. 71704170) supported by National Natural Science Foundation of Chinathe project "Study on Regularity and Dynamics of Knowledge Diffusion among Scientific Disciplines" (No. 71704063) supported by National Natura Science Foundation of Chinathe Youth Innovation Promotion Association, CAS (Grant No. 2016159)
文摘Purpose: Formal concept analysis(FCA) and concept lattice theory(CLT) are introduced for constructing a network of IDR topics and for evaluating their effectiveness for knowledge structure exploration.Design/methodology/approach: We introduced the theory and applications of FCA and CLT, and then proposed a method for interdisciplinary knowledge discovery based on CLT. As an example of empirical analysis, interdisciplinary research(IDR) topics in Information & Library Science(LIS) and Medical Informatics, and in LIS and Geography-Physical, were utilized as empirical fields. Subsequently, we carried out a comparative analysis with two other IDR topic recognition methods.Findings: The CLT approach is suitable for IDR topic identification and predictions.Research limitations: IDR topic recognition based on the CLT is not sensitive to the interdisciplinarity of topic terms, since the data can only reflect whether there is a relationship between the discipline and the topic terms. Moreover, the CLT cannot clearly represent a large amounts of concepts.Practical implications: A deeper understanding of the IDR topics was obtained as the structural and hierarchical relationships between them were identified, which can help to get more precise identification and prediction to IDR topics.Originality/value: IDR topics identification based on CLT have performed well and this theory has several advantages for identifying and predicting IDR topics. First, in a concept lattice, there is a partial order relation between interconnected nodes, and consequently, a complete concept lattice can present hierarchical properties. Second, clustering analysis of IDR topics based on concept lattices can yield clusters that highlight the essential knowledge features and help display the semantic relationship between different IDR topics. Furthermore, the Hasse diagram automatically displays all the IDR topics associated with the different disciplines, thus forming clusters of specific concepts and visually retaining and presenting the associations of IDR topics through multiple inheritance relationships between the concepts.
文摘高价值专利识别是抢占产业全球战略高地、推动产业持续高效健康发展的重要课题,可为产业关键核心技术的挖掘提供重要线索。本文从专利技术距离测度视角出发,在进行主题聚类提取领域上位类主题基础上,提出一种基于主题知识贡献距离与主题联系程度双维影响下的高价值专利识别方法。在主题知识贡献距离维度上,构建专利间分层专利引用网络,计算各专利与主题的持续知识贡献值,基于知识贡献时间序列计算主题间的动态时间规整(dynamic time warping,DTW)距离,形成主题知识贡献距离矩阵;在主题联系程度维度上,构建主题与专利二分图网络,结合专利共现频率与引用关系强度进行初始强度与全局逻辑计算,形成主题联系程度矩阵。融合双维度矩阵构建专利技术距离矩阵,基于技术距离矩阵进行各专利的绝对技术距离计算,选取阈值范围内的高绝对技术距离专利作为领域内高技术价值的专利。经验证数据集检验,本文方法的精准率达到0.8218,F1指标达到0.8014。基于此,对“生成式人工智能”领域专利进行实证,识别出产业内具有较高价值的专利1437件,并发现识别出的高价值专利集具有较高的转让比例,转让比例达58.59%。本文基于技术本质的视角对专利间的技术差距进行量化,打破了以往仅从外部特征或简单统计数据判断专利价值的局限性,提升了识别的准确性;同时,提出双维度的技术距离影响机理,进一步提升了识别的可解释性。
文摘微信群组中存在大量会话文本数据,对其进行关键词提取有助于理解群组动态和主题演变。由于微信会话文本存在长度短、主题交叉、语言不规范等特点,传统提取方法效果欠佳。为此,提出了一个基于会话主题聚类的多阶段关键词提取算法。首先,提出了一种结合预训练知识的会话主题聚类算法(Single Pass Using Thread Segmentation and Pre-training Knowledge,SP_(TSPK)),综合考虑语义相关性、消息活跃度和用户亲密度,有效解决了会话主题交叉和信息量不足的问题。其次,设计了一种多阶段关键词提取算法(Multi-Stage Keyword Extraction,MSKE),将任务分解为无监督关键词抽取和有监督关键词生成,有效提取原文中存在和缺失的关键词,减少了候选词规模和语义冗余;最终,组合SP_(TSPK)算法与MSKE算法实现微信会话文本关键词提取。在WeChat数据集上相比AutoKeyGen算法,F_(1)@5和F_(1)@O平均提升了12.8%与10.8%,R@10平均达到其2.59倍。实验结果表明,该算法能有效地提取微信会话文本关键词。
文摘【背景】蛹虫草(Cordyceps militaris)作为虫草科虫草属的模式种一直受到全球研究人员的关注。【目的】多维度探讨蛹虫草研究的当前状况与未来趋势。【方法】基于Web of Science核心合集数据库对2005-2024年间有关蛹虫草的SCI核心集论文进行了全面的数据搜集、整理、分析和可视化处理。【结果】过去20年里,蛹虫草研究已从单一的培养特性拓展至跨学科领域,尤其是其活性成分和药理学效应已成为学术界关注的焦点。文献计量分析结果显示,2005-2009年间,主要研究方向为蛹虫草的人工培养。2010-2014年间,研究主题扩展至子实体相关的药理学,研究地位显著提升。2015年后,研究主题进一步多元化,涵盖了优化、表达、氧化应激、真菌、抗氧化剂、化学成分、NF-κB、细胞周期停滞等领域,显示了从培养技术向深入的生物学和医学机制研究的转变。【结论】蛹虫草的研究经历了从传统培养研究向多学科交叉的深刻变革,未来研究将更加侧重于活性成分的功能机制、生物活性物质的药理作用及潜在的医学应用,为蛹虫草的深入研究和开发利用提供科学依据。
文摘【背景】微生物浸出是处理尾矿、废弃矿石、低品位矿、难处理矿的有效浸出手段;相较于传统的浸出技术,其具有环境友好、收益好、冶金效率高等优点。【目的】深入探讨微生物浸出在全球范围内的发展趋势和学术影响,并协助研究人员确定研究方向,开展相关研究,了解该领域最相关的课题。【方法】基于Web of Science核心合集数据库对2011-2023年全球微生物浸出的文献进行检索和分析。【结果】年发文趋势揭示微生物浸出领域的研究热度有所下降。高被引文献的研究表明,除了文献本身的质量外,另一个重要因素是国家政策支持和资金的可用性。全球共有80个国家包括1546个机构开展研究,共发表在580种期刊上。中国、伊朗、印度和澳大利亚进行了大量的研究,集成了冶金工程、环境科学生态学、采矿工程和生物技术与应用微生物学等多学科。聚类分析确定了4个经常出现的关键词:黄铜矿、废旧电路板、重金属和浸出,这为研究人员提供了新的检索词。【结论】目前对微生物浸出的研究主要集中在单个菌株,而混合菌株与矿物的吸附和耐受机理是未来发展所向。
