在对HDFS进行分析和研究的基础上,在HDFS文件分布式系统中应用File System API进行文件存储和访问,并通过改进的蚁群算法对副本选择进行优化。HDFS API能够有效完成海量数据的存储和管理,提高海量数据存储的效率。通过改进的蚁群算法提...在对HDFS进行分析和研究的基础上,在HDFS文件分布式系统中应用File System API进行文件存储和访问,并通过改进的蚁群算法对副本选择进行优化。HDFS API能够有效完成海量数据的存储和管理,提高海量数据存储的效率。通过改进的蚁群算法提升了文件读取时副本选择的效率,进一步提高了系统效率并使负载均衡。展开更多
This paper describes a method for building hot snapshot copy based on windows-file system （HSCF）. The architecture and running mechanism of HSCF are discussed after giving a comparison with other on-line backup tecb...This paper describes a method for building hot snapshot copy based on windows-file system （HSCF）. The architecture and running mechanism of HSCF are discussed after giving a comparison with other on-line backup tecbnology. HSCF, based on a file system filter driver, protects computer data and ensures their integrity and consistency with following three steps： access to open files, synchronization and copy on-write. Its strategies for improving system performance are analyzed including priority setting, incremental snapshot and load balance. HSCF is a new kind of snapshot technology to solve the data integrity and consistency problem in online backup, which is different from other storage-level snapshot and Open File Solution.展开更多
The bane of achieving a scalable distributed file sharing system is the centralized data system and single server oriented file [sharing] system. In this paper, the solution to the problems in a distributed environmen...The bane of achieving a scalable distributed file sharing system is the centralized data system and single server oriented file [sharing] system. In this paper, the solution to the problems in a distributed environment is presented. Thus, inability to upload sizeable files, slow transportation of files, weak security and lack of fault tolerance are the major problems in the existing system. Hence, the utmost need is to build a client-server network that runs on two or more server systems in order to implement scalability, such that when one server is down, the other(s) would still hold up the activities within the network. And to achieve this reliable network environment, LINUX network operating system is recommended among others as a preferred platform for the synchronization of the server systems, such that every user activity like sending of internal memos/mails, publication of academic articles, is replicated;thereby, achieving the proposed result. Hence, Scalable Distributed File Sharing System provides the robustness required to having a reliable intranet.展开更多
A new routing algorithm of peer-to-peer file sharing system with routing indices was proposed, in which a node forwards a query to neighbors that are more likely to have answers based on its statistics. The proposed a...A new routing algorithm of peer-to-peer file sharing system with routing indices was proposed, in which a node forwards a query to neighbors that are more likely to have answers based on its statistics. The proposed algorithm was tested by creating a P2P simulator and varying the input parameters, and was compared to the search algorithms using flooding （FLD） and random walk （RW）. The result shows that with the proposed design, the queries are muted effectively, the network flows are reduced remarkably, and the peer-to-peer file sharing system gains a good expansibility.展开更多
ABSTRACT: This paper generalizes the makeup and forming dynamic mechanism of natural disaster systems, principles and methods of comprehensive division of natural disasters, as well as structure, function and up-build...ABSTRACT: This paper generalizes the makeup and forming dynamic mechanism of natural disaster systems, principles and methods of comprehensive division of natural disasters, as well as structure, function and up-build routes of map and file information visualization system (MFIVS). Taking the Changjiang(Yangtze) Valley as an example, on the basis of revealing up the integrated mechanism on the formations of its natural disasters and its distributing law, thereafter, the paper relies on the MFIVS technique, adopts two top-down and bottom-up approaches to study a comprehensive division of natural disasters. It is relatively objective and precise that the required division results include three natural disaster sections and nine natural disaster sub-sections, which can not only provide a scientific basis for utilizing natural resources and controlling natural disaster and environmental degradation, but also be illuminated to a concise, practical and effective technique on comprehensive division.展开更多
In this paper, we explored a load-balancing algorithm in a cluster file system contains two levels of metadata-server, primary-level server quickly distributestasks to second-level servers depending on the closest loa...In this paper, we explored a load-balancing algorithm in a cluster file system contains two levels of metadata-server, primary-level server quickly distributestasks to second-level servers depending on the closest load-balancing information. At the same time, we explored a method which accurately reflect I/O traffic and storage of storage-node： computing the heat-value of file, according to which we realized a more logical storage allocation. According to the experiment result, we conclude that this new algorithm shortens the executing time of tasks and improves the system performance compared with other load algorithm.展开更多
Data layout in a file system is the organization of data stored in external storages. The data layout has a huge impact on performance of storage systems. We survey three main kinds of data layout in traditional file ...Data layout in a file system is the organization of data stored in external storages. The data layout has a huge impact on performance of storage systems. We survey three main kinds of data layout in traditional file systems： in-place update file system, log-structured file system, and copy-on-write file sys- tem. Each file system has its own strengths and weaknesses under different circumstances. We also include a recent us- age of persistent layout in a file system that combines both flash memory and byte- addressable non- volatile memory. With this survey, we conclude that persistent data layout in file systems may evolve dramatically in the era of emerging non-volatile memory.展开更多
File systems are fundamental for computers and devices with data storage units. They allow operating systems to understand and organize streams of bytes and obtain readable files from them. There are numerous file sys...File systems are fundamental for computers and devices with data storage units. They allow operating systems to understand and organize streams of bytes and obtain readable files from them. There are numerous file systems available in the industry, all with their own unique features. Understanding how these file systems work is essential for computer science students, but their complex nature can be difficult and challenging to grasp, especially for students at the beginning of their career. The Zion File System Simulator was designed with this in mind. Zion is a teaching and experimenting tool, in the form of a small application, built to help students understand how the I/O manager of an operating system interacts with the drive through the file system. Users can see and analyze the structure of a simple, flat file system provided with Zion, or simulate the most common structures such as FAT or NTFS. Students can also create their own implementations and run them through the simulator to analyze the different behaviors. Zion runs on Windows, and the application is provided with dynamic-link libraries that include the interfaces of a file system and a volume manager. These interfaces allow programmers to build their own file system or volume manager in Visual Studio using any .NET language (3.0 or above). Zion gives the users the power to adjust simulated architectural parameters such as volume and block size, or performance factors such as seek and transfer time. Zion runs workloads of I/O operations such as “create,” “delete,” “read,” and “write,” and analyzes the resulting metrics including I/O operations, read/write time, and disk fragmentation. Zion is a learning tool. It is not designed for measuring accurate performance of file systems and volume managers. The robustness of the application, together with its expandability, makes Zion a potential laboratory tool for computer science classes, helping students learn how file systems work and interact with an operating system.展开更多
IIn order to improve the performance of wireless distributed peer-to-peer(P2P)files sharing systems,a general system architecture and a novel peer selecting model based on fuzzy cognitive maps(FCM)are proposed in this...IIn order to improve the performance of wireless distributed peer-to-peer(P2P)files sharing systems,a general system architecture and a novel peer selecting model based on fuzzy cognitive maps(FCM)are proposed in this paper.The new model provides an effective approach on choosing an optimal peer from several resource discovering results for the best file transfer.Compared with the traditional min-hops scheme that uses hops as the only selecting criterion,the proposed model uses FCM to investigate the complex relationships among various relative factors in wireless environments and gives an overall evaluation score on the candidate.It also has strong scalability for being independent of specified P2P resource discovering protocols.Furthermore,a complete implementation is explained in concrete modules.The simulation results show that the proposed model is effective and feasible compared with min-hops scheme,with the success transfer rate increased by at least 20% and transfer time improved as high as 34%.展开更多
Working with files and the safety of information has always been relevant, especially in financial institutions where the requirements for the safety of information and security are especially important. And in today...Working with files and the safety of information has always been relevant, especially in financial institutions where the requirements for the safety of information and security are especially important. And in today’s conditions, when an earthquake can destroy the floor of a city in an instant, or when a missile hits an office and all servers turn into scrap metal, the issue of data safety becomes especially important. Also, you can’t put the cost of the software and the convenience of working with files in last place. Especially if an office worker needs to find the necessary information on a client, a financial contract or a company’s financial product in a few seconds. Also, during the operation of computer equipment, failures are possible, and some of them can lead to partial or complete loss of information. In this paper, it is proposed to create another level of abstraction for working with the file system, which will be based on a relational database as a storage of objects and access rights to objects. Also considered are possible protocols for transferring data to other programs that work with files, these can be both small sites and the operating system itself. This article will be especially interesting for financial institutions or companies operating in the banking sector. The purpose of this article is an attempt to introduce another level of abstraction for working with files. A level that is completely abstracted from the storage medium.展开更多
In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of ...In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.展开更多
Satellite networking communications in navigation satellite system and spacebased deep space exploration have the features of a long delay and high bit error rate （BER）. Through analyzing the advantages and disadvan...Satellite networking communications in navigation satellite system and spacebased deep space exploration have the features of a long delay and high bit error rate （BER）. Through analyzing the advantages and disadvantages of the Consulta tive Committee for the Space Data System （CCSDS） file delivery protocol （CFDP）, a new improved repeated sending file delivery protocol （RSFDP） based on the adaptive repeated sending is put forward to build an efficient and reliable file transmission. According to the estimation of the BER of the transmission link, RSFDP repeatedly sends the lost protocol data units （PDUs） at the stage of the retransmission to improve the success rate and reduce time of the retransmission. Theoretical analyses and results of the Opnet simulation indicate that the performance of RSFDP has significant improvement gains over CFDP in the link with a long delay and high BER. The realizing results based on the space borne filed programmable gate array （FPGA） platform show the applicability of the proposed algorithm.展开更多
The large scale and distribution of cloud computing storage have become the major challenges in cloud forensics for file extraction. Current disk forensic methods do not adapt to cloud computing well and the forensic ...The large scale and distribution of cloud computing storage have become the major challenges in cloud forensics for file extraction. Current disk forensic methods do not adapt to cloud computing well and the forensic research on distributed file system is inadequate. To address the forensic problems, this paper uses the Hadoop distributed file system （HDFS） as a case study and proposes a forensic method for efficient file extraction based on three-level （3L） mapping. First, HDFS is analyzed from overall architecture to local file system. Second, the 3L mapping of an HDFS file from HDFS namespace to data blocks on local file system is established and a recovery method for deleted files based on 3L mapping is presented. Third, a multi-node Hadoop framework via Xen virtualization platform is set up to test the performance of the method. The results indicate that the proposed method could succeed in efficient location of large files stored across data nodes, make selective image of disk data and get high recovery rate of deleted files.展开更多
文摘在对HDFS进行分析和研究的基础上,在HDFS文件分布式系统中应用File System API进行文件存储和访问,并通过改进的蚁群算法对副本选择进行优化。HDFS API能够有效完成海量数据的存储和管理,提高海量数据存储的效率。通过改进的蚁群算法提升了文件读取时副本选择的效率,进一步提高了系统效率并使负载均衡。
基金Supported by the National Natural Science Foun-dation of China (60473023) National Innovation Foundation forSmall Technology Based Firms(04C26214201280)
文摘This paper describes a method for building hot snapshot copy based on windows-file system （HSCF）. The architecture and running mechanism of HSCF are discussed after giving a comparison with other on-line backup tecbnology. HSCF, based on a file system filter driver, protects computer data and ensures their integrity and consistency with following three steps： access to open files, synchronization and copy on-write. Its strategies for improving system performance are analyzed including priority setting, incremental snapshot and load balance. HSCF is a new kind of snapshot technology to solve the data integrity and consistency problem in online backup, which is different from other storage-level snapshot and Open File Solution.
文摘The bane of achieving a scalable distributed file sharing system is the centralized data system and single server oriented file [sharing] system. In this paper, the solution to the problems in a distributed environment is presented. Thus, inability to upload sizeable files, slow transportation of files, weak security and lack of fault tolerance are the major problems in the existing system. Hence, the utmost need is to build a client-server network that runs on two or more server systems in order to implement scalability, such that when one server is down, the other(s) would still hold up the activities within the network. And to achieve this reliable network environment, LINUX network operating system is recommended among others as a preferred platform for the synchronization of the server systems, such that every user activity like sending of internal memos/mails, publication of academic articles, is replicated;thereby, achieving the proposed result. Hence, Scalable Distributed File Sharing System provides the robustness required to having a reliable intranet.
文摘A new routing algorithm of peer-to-peer file sharing system with routing indices was proposed, in which a node forwards a query to neighbors that are more likely to have answers based on its statistics. The proposed algorithm was tested by creating a P2P simulator and varying the input parameters, and was compared to the search algorithms using flooding （FLD） and random walk （RW）. The result shows that with the proposed design, the queries are muted effectively, the network flows are reduced remarkably, and the peer-to-peer file sharing system gains a good expansibility.
基金Under the auspices of President Foundation of the Chinese Academy of Sciences(1999).
文摘ABSTRACT: This paper generalizes the makeup and forming dynamic mechanism of natural disaster systems, principles and methods of comprehensive division of natural disasters, as well as structure, function and up-build routes of map and file information visualization system (MFIVS). Taking the Changjiang(Yangtze) Valley as an example, on the basis of revealing up the integrated mechanism on the formations of its natural disasters and its distributing law, thereafter, the paper relies on the MFIVS technique, adopts two top-down and bottom-up approaches to study a comprehensive division of natural disasters. It is relatively objective and precise that the required division results include three natural disaster sections and nine natural disaster sub-sections, which can not only provide a scientific basis for utilizing natural resources and controlling natural disaster and environmental degradation, but also be illuminated to a concise, practical and effective technique on comprehensive division.
基金Supported by the Industrialized Foundation ofHebei Province(020501) the Natural Science Foundation of HebeiUniversity(2005Q04)
文摘In this paper, we explored a load-balancing algorithm in a cluster file system contains two levels of metadata-server, primary-level server quickly distributestasks to second-level servers depending on the closest load-balancing information. At the same time, we explored a method which accurately reflect I/O traffic and storage of storage-node： computing the heat-value of file, according to which we realized a more logical storage allocation. According to the experiment result, we conclude that this new algorithm shortens the executing time of tasks and improves the system performance compared with other load algorithm.
基金supported by ZTE Industry-Academia-Research Cooperation Funds
文摘Data layout in a file system is the organization of data stored in external storages. The data layout has a huge impact on performance of storage systems. We survey three main kinds of data layout in traditional file systems： in-place update file system, log-structured file system, and copy-on-write file sys- tem. Each file system has its own strengths and weaknesses under different circumstances. We also include a recent us- age of persistent layout in a file system that combines both flash memory and byte- addressable non- volatile memory. With this survey, we conclude that persistent data layout in file systems may evolve dramatically in the era of emerging non-volatile memory.
文摘File systems are fundamental for computers and devices with data storage units. They allow operating systems to understand and organize streams of bytes and obtain readable files from them. There are numerous file systems available in the industry, all with their own unique features. Understanding how these file systems work is essential for computer science students, but their complex nature can be difficult and challenging to grasp, especially for students at the beginning of their career. The Zion File System Simulator was designed with this in mind. Zion is a teaching and experimenting tool, in the form of a small application, built to help students understand how the I/O manager of an operating system interacts with the drive through the file system. Users can see and analyze the structure of a simple, flat file system provided with Zion, or simulate the most common structures such as FAT or NTFS. Students can also create their own implementations and run them through the simulator to analyze the different behaviors. Zion runs on Windows, and the application is provided with dynamic-link libraries that include the interfaces of a file system and a volume manager. These interfaces allow programmers to build their own file system or volume manager in Visual Studio using any .NET language (3.0 or above). Zion gives the users the power to adjust simulated architectural parameters such as volume and block size, or performance factors such as seek and transfer time. Zion runs workloads of I/O operations such as “create,” “delete,” “read,” and “write,” and analyzes the resulting metrics including I/O operations, read/write time, and disk fragmentation. Zion is a learning tool. It is not designed for measuring accurate performance of file systems and volume managers. The robustness of the application, together with its expandability, makes Zion a potential laboratory tool for computer science classes, helping students learn how file systems work and interact with an operating system.
基金Sponsored by the National Natural Science Foundation of China(Grant No.60672124 and 60832009)Hi-Tech Research and Development Program(National 863 Program)(Grant No.2007AA01Z221)
文摘IIn order to improve the performance of wireless distributed peer-to-peer(P2P)files sharing systems,a general system architecture and a novel peer selecting model based on fuzzy cognitive maps(FCM)are proposed in this paper.The new model provides an effective approach on choosing an optimal peer from several resource discovering results for the best file transfer.Compared with the traditional min-hops scheme that uses hops as the only selecting criterion,the proposed model uses FCM to investigate the complex relationships among various relative factors in wireless environments and gives an overall evaluation score on the candidate.It also has strong scalability for being independent of specified P2P resource discovering protocols.Furthermore,a complete implementation is explained in concrete modules.The simulation results show that the proposed model is effective and feasible compared with min-hops scheme,with the success transfer rate increased by at least 20% and transfer time improved as high as 34%.
文摘Working with files and the safety of information has always been relevant, especially in financial institutions where the requirements for the safety of information and security are especially important. And in today’s conditions, when an earthquake can destroy the floor of a city in an instant, or when a missile hits an office and all servers turn into scrap metal, the issue of data safety becomes especially important. Also, you can’t put the cost of the software and the convenience of working with files in last place. Especially if an office worker needs to find the necessary information on a client, a financial contract or a company’s financial product in a few seconds. Also, during the operation of computer equipment, failures are possible, and some of them can lead to partial or complete loss of information. In this paper, it is proposed to create another level of abstraction for working with the file system, which will be based on a relational database as a storage of objects and access rights to objects. Also considered are possible protocols for transferring data to other programs that work with files, these can be both small sites and the operating system itself. This article will be especially interesting for financial institutions or companies operating in the banking sector. The purpose of this article is an attempt to introduce another level of abstraction for working with files. A level that is completely abstracted from the storage medium.
基金supported by the Research Fund of National Key Laboratory of Computer Architecture under Grant No.CARCH201501the Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing under Grant No.2016A09
文摘In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.
基金supported by the National High Technology Research and Development Program of China (863 Program) (2011AA1569)
文摘Satellite networking communications in navigation satellite system and spacebased deep space exploration have the features of a long delay and high bit error rate （BER）. Through analyzing the advantages and disadvantages of the Consulta tive Committee for the Space Data System （CCSDS） file delivery protocol （CFDP）, a new improved repeated sending file delivery protocol （RSFDP） based on the adaptive repeated sending is put forward to build an efficient and reliable file transmission. According to the estimation of the BER of the transmission link, RSFDP repeatedly sends the lost protocol data units （PDUs） at the stage of the retransmission to improve the success rate and reduce time of the retransmission. Theoretical analyses and results of the Opnet simulation indicate that the performance of RSFDP has significant improvement gains over CFDP in the link with a long delay and high BER. The realizing results based on the space borne filed programmable gate array （FPGA） platform show the applicability of the proposed algorithm.
基金Supported by the National High Technology Research and Development Program of China(863 Program)(2015AA016006)the National Natural Science Foundation of China(60903220)
文摘The large scale and distribution of cloud computing storage have become the major challenges in cloud forensics for file extraction. Current disk forensic methods do not adapt to cloud computing well and the forensic research on distributed file system is inadequate. To address the forensic problems, this paper uses the Hadoop distributed file system （HDFS） as a case study and proposes a forensic method for efficient file extraction based on three-level （3L） mapping. First, HDFS is analyzed from overall architecture to local file system. Second, the 3L mapping of an HDFS file from HDFS namespace to data blocks on local file system is established and a recovery method for deleted files based on 3L mapping is presented. Third, a multi-node Hadoop framework via Xen virtualization platform is set up to test the performance of the method. The results indicate that the proposed method could succeed in efficient location of large files stored across data nodes, make selective image of disk data and get high recovery rate of deleted files.
