Web crawlers are an important part of modern search engines.With the development of the times,data has exploded and humans have entered a“big data era”.For example,Wikipedia carries the knowledge from all over the w...Web crawlers are an important part of modern search engines.With the development of the times,data has exploded and humans have entered a“big data era”.For example,Wikipedia carries the knowledge from all over the world,records the realtime news that occurs every day,and provides users with a good database of data,but because of the large amount of data,it puts a lot of pressure on users to search.At present,single-threaded crawling data can no longer meet the requirements of text crawling.In order to improve the performance and program versatility of single-threaded crawlers,a high-speed multi-threaded web crawler is designed to crawl the network hyper-scale text database.Multi-threaded crawling uses multiple threads to process web pages in parallel,combining breadth-first and depth-first algorithms to control web crawling.The practice project is based on the Python language to achieve multi-threaded optimization network hyper-large-scale text database-Wikipedia book crawling method,the project is inspired by the article on the Wikipedia article in the Big Data Digest public number.展开更多
This paper introduces the general process of the search algorithm Structure through the knight problem. According to the characteristics of the problem, we detailed discuss the DFS（Depth First Search） algorithm and ...This paper introduces the general process of the search algorithm Structure through the knight problem. According to the characteristics of the problem, we detailed discuss the DFS（Depth First Search） algorithm and BFS（Breadth First Search） algorithm, and combine the two algorithms together to solve the knights coverage problem. This article has a good reference for the mixed-use scenarios which requires a variety of search algorithms.展开更多
输电线路的安全稳定运行对现代社会的电力供应至关重要。传统的输电线路运维检修工作面临诸多挑战,包括人力资源需求大、作业环境复杂以及安全风险高等。近年来,无人机技术为输电线路运维检修提供了一种有效的解决方案。文章通过构建输...输电线路的安全稳定运行对现代社会的电力供应至关重要。传统的输电线路运维检修工作面临诸多挑战,包括人力资源需求大、作业环境复杂以及安全风险高等。近年来,无人机技术为输电线路运维检修提供了一种有效的解决方案。文章通过构建输电线路节点模拟图,采用深度优先搜索(Depth First Search,DFS)算法进行路径规划,并结合环境感知技术和避障算法,实现无人机在输电线路运维检修中的自主化作业。通过实验验证,所提方法能够有效规划无人机的巡航路径,避开设定的障碍物,完成对输电线路的全面巡检任务,达到提高运维效率、降低成本以及减少安全风险的目的。展开更多
电力系统仿真验证往往希望通过拓扑结构图直观地分析网络的潮流分布以及动态特性。然而电力系统机电暂态过程仿真软件如BPA、PSS/E和PSASP都不能自动地根据电力系统的电气联系合理地布置网络中的元件,而需要人为地调整各元件的位置来形...电力系统仿真验证往往希望通过拓扑结构图直观地分析网络的潮流分布以及动态特性。然而电力系统机电暂态过程仿真软件如BPA、PSS/E和PSASP都不能自动地根据电力系统的电气联系合理地布置网络中的元件,而需要人为地调整各元件的位置来形成一个直观的电气接线图。这种人为手动调整,不仅给仿真增加了工作量,更有可能带来更多的人为误差。为此,文中提出了基于图论的深度优先搜索(depth first searching,DFS)算法,依据电力系统的电气拓扑结构形成电力系统生成树的实现方法。用文中方法生成的IEEE9节点算例系统的可视化界面验证了该算法的有效性和准确性。展开更多
实现各类预想故障下潮流转移比快速仿真分析是电网安全稳定运行的重要保证。针对现有实际运行方式中潮流转移分析困难问题,提出大规模电力系统潮流转移比多核并行批处理方法。该方法基于广泛使用的商业大系统分析工具,在参数解析分类、...实现各类预想故障下潮流转移比快速仿真分析是电网安全稳定运行的重要保证。针对现有实际运行方式中潮流转移分析困难问题,提出大规模电力系统潮流转移比多核并行批处理方法。该方法基于广泛使用的商业大系统分析工具,在参数解析分类、故障自动设置及结果解析的基础上,引入深度优先搜索(depth first search,DFS)算法进行孤立节点和孤岛区域检测以保证网络完整性,结合潮流计算合理性的自动判别以实现潮流转移比的批处理分析;同时在多核环境下,构建基于Fork/Join的并行框架,采用"分治模式"递归分解计算任务,从而实现分析方法的多核并行。算例仿真和在云南电网的实际应用验证了所提方法的有效性和快速性。展开更多
基金This research is funded by the Open Foundation for the University Innovation Platform in the Hunan Province,grant number 16K013Hunan Provincial Natural Science Foundation of China,grant number 2017JJ2016+2 种基金2016 Science Research Project of Hunan Provincial Department of Education,grant number 16C0269.Accurate crawler design and implementation with a data cleaning function,National Students innovation and entrepreneurship of training program,grant number 201811532010.This research work is implemented at the 2011 Collaborative Innovation Center for Development and Utilization of Finance and Economics Big Data Property,Universities of Hunan Province.Open Foundation for the University Innovation Platform in the Hunan Province,grant number 16K013Hunan Provincial Natural Science Foundation of China,grant number 2017JJ20162016 Science Research Project of Hunan Provincial Department of Education,grant number 16C0269.This research work is implemented at the 2011 Collaborative Innovation Center for Development and Utilization of Finance and Economics Big Data Property,Universities of Hunan Province.Open project,grant number 20181901CRP03,20181901CRP04,20181901CRP05.
文摘Web crawlers are an important part of modern search engines.With the development of the times,data has exploded and humans have entered a“big data era”.For example,Wikipedia carries the knowledge from all over the world,records the realtime news that occurs every day,and provides users with a good database of data,but because of the large amount of data,it puts a lot of pressure on users to search.At present,single-threaded crawling data can no longer meet the requirements of text crawling.In order to improve the performance and program versatility of single-threaded crawlers,a high-speed multi-threaded web crawler is designed to crawl the network hyper-scale text database.Multi-threaded crawling uses multiple threads to process web pages in parallel,combining breadth-first and depth-first algorithms to control web crawling.The practice project is based on the Python language to achieve multi-threaded optimization network hyper-large-scale text database-Wikipedia book crawling method,the project is inspired by the article on the Wikipedia article in the Big Data Digest public number.
文摘This paper introduces the general process of the search algorithm Structure through the knight problem. According to the characteristics of the problem, we detailed discuss the DFS（Depth First Search） algorithm and BFS（Breadth First Search） algorithm, and combine the two algorithms together to solve the knights coverage problem. This article has a good reference for the mixed-use scenarios which requires a variety of search algorithms.
文摘输电线路的安全稳定运行对现代社会的电力供应至关重要。传统的输电线路运维检修工作面临诸多挑战,包括人力资源需求大、作业环境复杂以及安全风险高等。近年来,无人机技术为输电线路运维检修提供了一种有效的解决方案。文章通过构建输电线路节点模拟图,采用深度优先搜索(Depth First Search,DFS)算法进行路径规划,并结合环境感知技术和避障算法,实现无人机在输电线路运维检修中的自主化作业。通过实验验证,所提方法能够有效规划无人机的巡航路径,避开设定的障碍物,完成对输电线路的全面巡检任务,达到提高运维效率、降低成本以及减少安全风险的目的。
文摘电力系统仿真验证往往希望通过拓扑结构图直观地分析网络的潮流分布以及动态特性。然而电力系统机电暂态过程仿真软件如BPA、PSS/E和PSASP都不能自动地根据电力系统的电气联系合理地布置网络中的元件,而需要人为地调整各元件的位置来形成一个直观的电气接线图。这种人为手动调整,不仅给仿真增加了工作量,更有可能带来更多的人为误差。为此,文中提出了基于图论的深度优先搜索(depth first searching,DFS)算法,依据电力系统的电气拓扑结构形成电力系统生成树的实现方法。用文中方法生成的IEEE9节点算例系统的可视化界面验证了该算法的有效性和准确性。
文摘实现各类预想故障下潮流转移比快速仿真分析是电网安全稳定运行的重要保证。针对现有实际运行方式中潮流转移分析困难问题,提出大规模电力系统潮流转移比多核并行批处理方法。该方法基于广泛使用的商业大系统分析工具,在参数解析分类、故障自动设置及结果解析的基础上,引入深度优先搜索(depth first search,DFS)算法进行孤立节点和孤岛区域检测以保证网络完整性,结合潮流计算合理性的自动判别以实现潮流转移比的批处理分析;同时在多核环境下,构建基于Fork/Join的并行框架,采用"分治模式"递归分解计算任务,从而实现分析方法的多核并行。算例仿真和在云南电网的实际应用验证了所提方法的有效性和快速性。
