Fraction repetition(FR)codes are integral in distributed storage systems(DSS)with exact repair-by-transfer,while pliable fraction repetition codes are vital for DSSs in which both the per-node storage and repetition d...Fraction repetition(FR)codes are integral in distributed storage systems(DSS)with exact repair-by-transfer,while pliable fraction repetition codes are vital for DSSs in which both the per-node storage and repetition degree can easily be adjusted simultaneously.This paper introduces a new type of pliable FR codes,called absolute balanced pliable FR(ABPFR)codes,in which the access balancing in DSS is considered.Additionally,the equivalence between pliable FR codes and resolvable transversal packings in combinatorial design theory is presented.Then constructions of pliable FR codes and ABPFR codes based on resolvable transversal packings are presented.展开更多
To reduce the time required to complete the regeneration process of erasure codes, we propose a Tree-structured Parallel Regeneration (TPR) scheme for multiple data losses in distributed storage systems. Under the sch...To reduce the time required to complete the regeneration process of erasure codes, we propose a Tree-structured Parallel Regeneration (TPR) scheme for multiple data losses in distributed storage systems. Under the scheme, two algorithms are proposed for the construction of multiple regeneration trees, namely the edge-disjoint algorithm and edge-sharing algorithm. The edge-disjoint algorithm constructs multiple independent trees, and is simple and appropriate for environments where newcomers and their providers are distributed over a large area and have few intersections. The edge-sharing algorithm constructs multiple trees that compete to utilize the bandwidth, and make a better utilization of the bandwidth, although it needs to measure the available band-width and deal with the bandwidth changes; it is therefore difficult to implement in practical systems. The parallel regeneration for multiple data losses of TPR primarily includes two optimizations: firstly, transferring the data through the bandwidth optimized-paths in a pipe-line manner; secondly, executing data regeneration over multiple trees in parallel. To evaluate the proposal, we implement an event-based simulator and make a detailed comparison with some popular regeneration methods. The quantitative comparison results show that the use of TPR employing either the edge-disjoint algorithm or edge-sharing algorithm reduces the regeneration time significantly.展开更多
This paper describes the architecture of global distributed storage system for data grid. It focue on the management and the capability for the maximum users and maximum resources on the Internet, as well as performan...This paper describes the architecture of global distributed storage system for data grid. It focue on the management and the capability for the maximum users and maximum resources on the Internet, as well as performance and other issues.展开更多
Remote data auditing becomes critical to ensure the storage reliability in distributed cloud storage.Recently,Le et al proposed an efficient private data auditing scheme NC-Audit designed for regenerating codes,which ...Remote data auditing becomes critical to ensure the storage reliability in distributed cloud storage.Recently,Le et al proposed an efficient private data auditing scheme NC-Audit designed for regenerating codes,which claimed that NC-Audit can effectively realize privacy-preserving data auditing for distributed storage systems.However,our analysis shows that NC-Audit is not secure for that the adversarial cloud can forge some illegal blocks to cheat the auditor successfully with a high probability even without storing the user’s whole data,when the coding field is large enough.展开更多
Erasure code is widely used as the redundancy scheme in distributed storage system. When a storage node fails, the repair process often requires to transfer a large amount of data. Regenerating code and hierarchical c...Erasure code is widely used as the redundancy scheme in distributed storage system. When a storage node fails, the repair process often requires to transfer a large amount of data. Regenerating code and hierarchical code are two classes of codes proposed to reduce the repair bandwidth cost. Regenerating codes reduce the amount of data transferred by each helping node, while hierarchical codes reduce the number of nodes participating in the repair process. In this paper, we propose a "sub-code nesting framework" to combine them together. The resulting regenerating hierarchical code has low repair degree as hierarchical code and lower repair cost than hierarchical code. Our code can achieve exact regeneration of the failed node, and has the additional property of low updating complexity.展开更多
Current distributed parallel file systems and database systems can not satisfy the demands of data-intensive applications, such as storage capacity, access performance, reliability, scalability, and so on. Cluster-bas...Current distributed parallel file systems and database systems can not satisfy the demands of data-intensive applications, such as storage capacity, access performance, reliability, scalability, and so on. Cluster-based storage sys tems have some shortcomings, too. To solve this kind of problems, a novel PC storage cluster solution is proposed, a distributed storage system based on 3-tiered agent architecture is designed, the system reliability model based on the masterslave backup mode is built, and the system availability is analyzed with the Markov model. According to the system availability formula and the values of the system parameters, the novel system can provide higher reliability and availability to satisfy users＇ requirements,展开更多
Recently,research on a distributed storage system that efficiently manages a large amount of data has been actively conducted following data production and demand increase.Physical expansion limits exist for tradition...Recently,research on a distributed storage system that efficiently manages a large amount of data has been actively conducted following data production and demand increase.Physical expansion limits exist for traditional standalone storage systems,such as I/O and file system capacity.However,the existing distributed storage system does not consider where data is consumed and is more focused on data dissemination and optimizing the lookup cost of data location.And this leads to system performance degradation due to low locality occurring in a Wide Area Network(WAN)environment with high network latency.This problem hinders deploying distributed storage systems to multiple data centers over WAN.It lowers the scalability of distributed storage systems to accommodate data storage needs.This paper proposes a method for distributing data in a WAN environment considering network latency and data locality to solve this problem and increase overall system performance.The proposed distributed storage method monitors data utilization and locality to classify data temperature as hot,warm,and cold.With assigned data temperature,the proposed algorithm adaptively selects the appropriate data center and places data accordingly to overcome the excess latency from the WAN environment,leading to overall system performance degradation.This paper also conducts simulations to evaluate the proposed and existing distributed storage methods.The result shows that our proposed method reduced latency by 38%compared to the existing method.Therefore,the proposed method in this paper can be used in large-scale distributed storage systems over a WAN environment to improve latency and performance compared to existing methods,such as consistent hashing.展开更多
Distributed storage can store data in multiple devices or servers to improve data security.However,in today’s explosive growth of network data,traditional distributed storage scheme is faced with some severe challeng...Distributed storage can store data in multiple devices or servers to improve data security.However,in today’s explosive growth of network data,traditional distributed storage scheme is faced with some severe challenges such as insufficient performance,data tampering,and data lose.A distributed storage scheme based on blockchain has been proposed to improve security and efficiency of traditional distributed storage.Under this scheme,the following improvements have been made in this paper.This paper first analyzes the problems faced by distributed storage.Then proposed to build a new distributed storage blockchain scheme with sharding blockchain.The proposed scheme realizes the partitioning of the network and nodes by means of blockchain sharding technology,which can improve the efficiency of data verification between nodes.In addition,this paper uses polynomial commitment to construct a new verifiable secret share scheme called PolyVSS.This new scheme is one of the foundations for building our improved distributed storage blockchain scheme.Compared with the previous scheme,our new scheme does not require a trusted third party and has some new features such as homomorphic and batch opening.The security of VSS can be further improved.Experimental comparisons show that the proposed scheme significantly reduces storage and communication costs.展开更多
To ensure the reliability and availability of data,redundancy strategies are always required for distributed storage systems.Erasure coding,one of the representative redundancy strategies,has the advantage of low stor...To ensure the reliability and availability of data,redundancy strategies are always required for distributed storage systems.Erasure coding,one of the representative redundancy strategies,has the advantage of low storage overhead,which facilitates its employment in distributed storage systems.Among the various erasure coding schemes,XOR-based erasure codes are becoming popular due to their high computing speed.When a single-node failure occurs in such coding schemes,a process called data recovery takes place to retrieve the failed node’s lost data from surviving nodes.However,data transmission during the data recovery process usually requires a considerable amount of time.Current research has focused mainly on reducing the amount of data needed for data recovery to reduce the time required for data transmission,but it has encountered problems such as significant complexity and local optima.In this paper,we propose a random search recovery algorithm,named SA-RSR,to speed up single-node failure recovery of XOR-based erasure codes.SA-RSR uses a simulated annealing technique to search for an optimal recovery solution that reads and transmits a minimum amount of data.In addition,this search process can be done in polynomial time.We evaluate SA-RSR with a variety of XOR-based erasure codes in simulations and in a real storage system,Ceph.Experimental results in Ceph show that SA-RSR reduces the amount of data required for recovery by up to 30.0%and improves the performance of data recovery by up to 20.36%compared to the conventional recovery method.展开更多
With the advent of the era of big data,cloud computing,Internet of things,and other information industries continue to develop.There is an increasing amount of unstructured data such as pictures,audio,and video on the...With the advent of the era of big data,cloud computing,Internet of things,and other information industries continue to develop.There is an increasing amount of unstructured data such as pictures,audio,and video on the Internet.And the distributed object storage system has become the mainstream cloud storage solution.With the increasing number of distributed applications,data security in the distributed object storage system has become the focus.For the distributed object storage system,traditional defenses are means that fix discovered system vulnerabilities and backdoors by patching,or means to modify the corresponding structure and upgrade.However,these two kinds of means are hysteretic and hardly deal with unknown security threats.Based on mimic defense theory,this paper constructs the principle framework of the distributed object storage system and introduces the dynamic redundancy and heterogeneous function in the distributed object storage system architecture,which increases the attack cost,and greatly improves the security and availability of data.展开更多
Based on the energy storage cloud platform architecture,this study considers the extensive configuration of energy storage devices and the future large-scale application of electric vehicles at the customer side to bu...Based on the energy storage cloud platform architecture,this study considers the extensive configuration of energy storage devices and the future large-scale application of electric vehicles at the customer side to build a new mode of smart power consumption with a flexible interaction,smooth the peak/valley difference of the load side power,and improve energy efficiency.A plug and play device for customer-side energy storage and an internet-based energy storage cloud platform are developed herein to build a new intelligent power consumption mode with a flexible interaction suitable for ordinary customers.Based on the load perception of the power grid,this study aims to investigate the operating state and service life of distributed energy storage devices.By selecting an integrated optimal control scheme,this study designs a kind of energy optimization and deployment strategy for stratified partition to reduce the operating cost of the energy storage device on the client side.The effectiveness of the system and the control strategy is verified through the Suzhou client-side distributed energy storage demonstration project.展开更多
In distributed cloud storage systems, inevitably there exist multiple node failures at the same time. The existing methods of regenerating codes, including minimum storage regenerating(MSR) codes and minimum bandwidth...In distributed cloud storage systems, inevitably there exist multiple node failures at the same time. The existing methods of regenerating codes, including minimum storage regenerating(MSR) codes and minimum bandwidth regenerating(MBR) codes, are mainly to repair one single or several failed nodes, unable to meet the repair need of distributed cloud storage systems. In this paper, we present locally minimum storage regenerating(LMSR) codes to recover multiple failed nodes at the same time. Specifically, the nodes in distributed cloud storage systems are divided into multiple local groups, and in each local group(4, 2) or(5, 3) MSR codes are constructed. Moreover, the grouping method of storage nodes and the repairing process of failed nodes in local groups are studied. Theoretical analysis shows that LMSR codes can achieve the same storage overhead as MSR codes. Furthermore, we verify by means of simulation that, compared with MSR codes, LMSR codes can reduce the repair bandwidth and disk I/O overhead effectively.展开更多
There is instability in the distributed energy storage cloud group end region on the power grid side.In order to avoid large-scale fluctuating charging and discharging in the power grid environment and make the capaci...There is instability in the distributed energy storage cloud group end region on the power grid side.In order to avoid large-scale fluctuating charging and discharging in the power grid environment and make the capacitor components showa continuous and stable charging and discharging state,a hierarchical time-sharing configuration algorithm of distributed energy storage cloud group end region on the power grid side based on multi-scale and multi feature convolution neural network is proposed.Firstly,a voltage stability analysis model based onmulti-scale and multi feature convolution neural network is constructed,and the multi-scale and multi feature convolution neural network is optimized based on Self-OrganizingMaps(SOM)algorithm to analyze the voltage stability of the cloud group end region of distributed energy storage on the grid side under the framework of credibility.According to the optimal scheduling objectives and network size,the distributed robust optimal configuration control model is solved under the framework of coordinated optimal scheduling at multiple time scales;Finally,the time series characteristics of regional power grid load and distributed generation are analyzed.According to the regional hierarchical time-sharing configuration model of“cloud”,“group”and“end”layer,the grid side distributed energy storage cloud group end regional hierarchical time-sharing configuration algorithm is realized.The experimental results show that after applying this algorithm,the best grid side distributed energy storage configuration scheme can be determined,and the stability of grid side distributed energy storage cloud group end region layered timesharing configuration can be improved.展开更多
The knowledge graph with relational abundant information has been widely used as the basic data support for the retrieval platforms.Image and text descriptions added to the knowledge graph enrich the node information,...The knowledge graph with relational abundant information has been widely used as the basic data support for the retrieval platforms.Image and text descriptions added to the knowledge graph enrich the node information,which accounts for the advantage of the multi-modal knowledge graph.In the field of cross-modal retrieval platforms,multi-modal knowledge graphs can help to improve retrieval accuracy and efficiency because of the abundant relational infor-mation provided by knowledge graphs.The representation learning method is sig-nificant to the application of multi-modal knowledge graphs.This paper proposes a distributed collaborative vector retrieval platform(DCRL-KG)using the multi-modal knowledge graph VisualSem as the foundation to achieve efficient and high-precision multimodal data retrieval.Firstly,use distributed technology to classify and store the data in the knowledge graph to improve retrieval efficiency.Secondly,this paper uses BabelNet to expand the knowledge graph through multi-ple filtering processes and increase the diversification of information.Finally,this paper builds a variety of retrieval models to achieve the fusion of retrieval results through linear combination methods to achieve high-precision language retrieval and image retrieval.The paper uses sentence retrieval and image retrieval experi-ments to prove that the platform can optimize the storage structure of the multi-modal knowledge graph and have good performance in multi-modal space.展开更多
In distributed storage systems,file access efficiency has an important impact on the real-time nature of information forensics.As a popular approach to improve file accessing efficiency,prefetching model can fetches d...In distributed storage systems,file access efficiency has an important impact on the real-time nature of information forensics.As a popular approach to improve file accessing efficiency,prefetching model can fetches data before it is needed according to the file access pattern,which can reduce the I/O waiting time and increase the system concurrency.However,prefetching model needs to mine the degree of association between files to ensure the accuracy of prefetching.In the massive small file situation,the sheer volume of files poses a challenge to the efficiency and accuracy of relevance mining.In this paper,we propose a massive files prefetching model based on LSTM neural network with cache transaction strategy to improve file access efficiency.Firstly,we propose a file clustering algorithm based on temporal locality and spatial locality to reduce the computational complexity.Secondly,we propose a definition of cache transaction according to files occurrence in cache instead of time-offset distance based methods to extract file block feature accurately.Lastly,we innovatively propose a file access prediction algorithm based on LSTM neural network which predict the file that have high possibility to be accessed.Experiments show that compared with the traditional LRU and the plain grouping methods,the proposed model notably increase the cache hit rate and effectively reduces the I/O wait time.展开更多
We study the construction of minimum bandwidth regenerating code with combinatorial design. At first, a method of constructing minimum storage regenerating （MBR） codes is presented, which can tolerate only one-node ...We study the construction of minimum bandwidth regenerating code with combinatorial design. At first, a method of constructing minimum storage regenerating （MBR） codes is presented, which can tolerate only one-node failure. Then, we give examples to explain the code. Finally, we discuss the case of repairing multiple nodes, and analyze the performance with an example.展开更多
The proliferation of Internet of Things(IoT)systems has resulted in the generation of substantial data,presenting new challenges in reliable storage and trustworthy sharing.Conventional distributed storage systems are...The proliferation of Internet of Things(IoT)systems has resulted in the generation of substantial data,presenting new challenges in reliable storage and trustworthy sharing.Conventional distributed storage systems are hindered by centralized management and lack traceability,while blockchain systems are limited by low capacity and high latency.To address these challenges,the present study investigates the reliable storage and trustworthy sharing of IoT data,and presents a novel system architecture that integrates on-chain and off-chain data manage systems.This architecture,integrating blockchain and distributed storage technologies,provides high-capacity,high-performance,traceable,and verifiable data storage and access.The on-chain system,built on Hyperledger Fabric,manages metadata,verification data,and permission information of the raw data.The off-chain system,implemented using IPFS Cluster,ensures the reliable storage and efficient access to massive files.A collaborative storage server is designed to integrate on-chain and off-chain operation interfaces,facilitating comprehensive data operations.We provide a unified access interface for user-friendly system interaction.Extensive testing validates the system’s reliability and stable performance.The proposed approach significantly enhances storage capacity compared to standalone blockchain systems.Rigorous reliability tests consistently yield positive outcomes.With average upload and download throughputs of roughly 20 and 30 MB/s,respectively,the system’s throughput surpasses the blockchain system by a factor of 4 to 18.展开更多
Vehicular Ad hoc Networks(VANETs)become a very crucial addition in the Intelligent Transportation System(ITS).It is challenging for a VANET system to provide security services and parallelly maintain high throughput b...Vehicular Ad hoc Networks(VANETs)become a very crucial addition in the Intelligent Transportation System(ITS).It is challenging for a VANET system to provide security services and parallelly maintain high throughput by utilizing limited resources.To overcome these challenges,we propose a blockchain-based Secured Cluster-based MAC(SCB-MAC)protocol.The nearby vehicles heading towards the same direction will form a cluster and each of the clusters has its blockchain to store and distribute the safety messages.The message which contains emergency information and requires Strict Delay Requirement(SDR)for transmission are called safety messages(SM).Cluster Members(CMs)sign SMs with their private keys while sending them to the blockchain to confirm authentication,integrity,and confidentiality of the message.A Certificate Authority(CA)is responsible for physical verification,key generation,and privacy preservation of the vehicles.We implemented a test scenario as proof of concept and tested the safety message transmission(SMT)protocol in a real-world platform.Computational and storage overhead analysis shows that the proposed protocol for SMT implements security,authentication,integrity,robustness,non-repudiation,etc.while maintaining the SDR.Messages that are less important compared to the SMs are called non-safety messages(NSM)and vehicles use RTS/CTS mechanism for NSM transmission.Numerical studies show that the proposed NSM transmission method maintains 6 times more throughput,2 times less delay and 125%less Packet Dropping Rate(PDR)than traditional MAC protocols.These results prove that the proposed protocol outperforms the traditional MAC protocols.展开更多
During the initial construction of the Puguang Gas Field,information infrastructure was built.Due to the absence of unified planning and deployment,however,many“isolated information islands”were formed in data syste...During the initial construction of the Puguang Gas Field,information infrastructure was built.Due to the absence of unified planning and deployment,however,many“isolated information islands”were formed in data systems,and the data resources cannot meet the construction requirements of intelligent gas field.In this paper,the status quo and problems of data resources in the Puguang Gas Field were analyzed,and a data resource sharing center was constructed according to the overall architecture design of the Puguang Intelligent Gas Field.Based on the architecture design of data resource sharing center,the overall construction conception of data resource sharing center was put forward and the business data model was designed.Finally,the integrated data collection,storage,calculation and utilization was realized by establishing data standard,combing data sources and designing data services,and then it was applied on site.The following research results were obtained.First,the data resource sharing center is an important foundation for the construction of this project,and its overall architecture is divided into three layers from bottom to top,i.e.,data specification and standard,data collection,storage,calculation and utilization,and data control.Second,the data resource sharing center achieves the one-time collection,centralized storage,shared use and unified management of exploration&development,gathering&purification,production&operation and safety&environmental protection data,and provides an important data base for the construction of business system of the intelligent gas field and a comprehensive,reliable and effective data support for the intelligent and mobile application in the Puguang Gas Field.展开更多
基金Supported in part by the National Key R&D Program of China(No.2020YFA0712300)NSFC(No.61872353)。
文摘Fraction repetition(FR)codes are integral in distributed storage systems(DSS)with exact repair-by-transfer,while pliable fraction repetition codes are vital for DSSs in which both the per-node storage and repetition degree can easily be adjusted simultaneously.This paper introduces a new type of pliable FR codes,called absolute balanced pliable FR(ABPFR)codes,in which the access balancing in DSS is considered.Additionally,the equivalence between pliable FR codes and resolvable transversal packings in combinatorial design theory is presented.Then constructions of pliable FR codes and ABPFR codes based on resolvable transversal packings are presented.
基金supported by the National Grand Fundamental Research of China (973 Program) under Grant No. 2011CB302601the National High Technology Research and Development of China (863 Program) under GrantNo. 2013AA01A213+2 种基金the National Natural Science Foundation of China under Grant No. 60873215the Natural Science Foundation for Distinguished Young Scholars of Hunan Province under Grant No. S2010J5050Specialized Research Fund for the Doctoral Program of Higher Education under Grant No. 20124307110015
文摘To reduce the time required to complete the regeneration process of erasure codes, we propose a Tree-structured Parallel Regeneration (TPR) scheme for multiple data losses in distributed storage systems. Under the scheme, two algorithms are proposed for the construction of multiple regeneration trees, namely the edge-disjoint algorithm and edge-sharing algorithm. The edge-disjoint algorithm constructs multiple independent trees, and is simple and appropriate for environments where newcomers and their providers are distributed over a large area and have few intersections. The edge-sharing algorithm constructs multiple trees that compete to utilize the bandwidth, and make a better utilization of the bandwidth, although it needs to measure the available band-width and deal with the bandwidth changes; it is therefore difficult to implement in practical systems. The parallel regeneration for multiple data losses of TPR primarily includes two optimizations: firstly, transferring the data through the bandwidth optimized-paths in a pipe-line manner; secondly, executing data regeneration over multiple trees in parallel. To evaluate the proposal, we implement an event-based simulator and make a detailed comparison with some popular regeneration methods. The quantitative comparison results show that the use of TPR employing either the edge-disjoint algorithm or edge-sharing algorithm reduces the regeneration time significantly.
文摘This paper describes the architecture of global distributed storage system for data grid. It focue on the management and the capability for the maximum users and maximum resources on the Internet, as well as performance and other issues.
基金Supported by the National Natural Science Foundation of China(61872088)the Science and Technology Plan Project of Xi’an(2020KJWL02,2017CGWL35)the China National Study Abroad Fund。
文摘Remote data auditing becomes critical to ensure the storage reliability in distributed cloud storage.Recently,Le et al proposed an efficient private data auditing scheme NC-Audit designed for regenerating codes,which claimed that NC-Audit can effectively realize privacy-preserving data auditing for distributed storage systems.However,our analysis shows that NC-Audit is not secure for that the adversarial cloud can forge some illegal blocks to cheat the auditor successfully with a high probability even without storing the user’s whole data,when the coding field is large enough.
基金Supported by 973 Project of China (No. 2012CB315803)Research Fund for the Doctoral Program of Higher Education of China (No. 20100002110033)Open research Fund of National Mobile Communications Research Laboratory, Southeast University (No. 2011D11)
文摘Erasure code is widely used as the redundancy scheme in distributed storage system. When a storage node fails, the repair process often requires to transfer a large amount of data. Regenerating code and hierarchical code are two classes of codes proposed to reduce the repair bandwidth cost. Regenerating codes reduce the amount of data transferred by each helping node, while hierarchical codes reduce the number of nodes participating in the repair process. In this paper, we propose a "sub-code nesting framework" to combine them together. The resulting regenerating hierarchical code has low repair degree as hierarchical code and lower repair cost than hierarchical code. Our code can achieve exact regeneration of the failed node, and has the additional property of low updating complexity.
基金Supported by the Industrialization Foundation ofHebei Province (020501)the Natural Science Foundation ofHebei University (2005Q04)
文摘Current distributed parallel file systems and database systems can not satisfy the demands of data-intensive applications, such as storage capacity, access performance, reliability, scalability, and so on. Cluster-based storage sys tems have some shortcomings, too. To solve this kind of problems, a novel PC storage cluster solution is proposed, a distributed storage system based on 3-tiered agent architecture is designed, the system reliability model based on the masterslave backup mode is built, and the system availability is analyzed with the Markov model. According to the system availability formula and the values of the system parameters, the novel system can provide higher reliability and availability to satisfy users＇ requirements,
基金This research was supported by the Chung-Ang University Graduate Research Scholarship in 2021.This study was carried out with the support of‘R&D Program for Forest Science Technology(Project No.2021338C10-2223-CD02)’provided by Korea Forest Service(Korea Forestry Promotion Institute).
文摘Recently,research on a distributed storage system that efficiently manages a large amount of data has been actively conducted following data production and demand increase.Physical expansion limits exist for traditional standalone storage systems,such as I/O and file system capacity.However,the existing distributed storage system does not consider where data is consumed and is more focused on data dissemination and optimizing the lookup cost of data location.And this leads to system performance degradation due to low locality occurring in a Wide Area Network(WAN)environment with high network latency.This problem hinders deploying distributed storage systems to multiple data centers over WAN.It lowers the scalability of distributed storage systems to accommodate data storage needs.This paper proposes a method for distributing data in a WAN environment considering network latency and data locality to solve this problem and increase overall system performance.The proposed distributed storage method monitors data utilization and locality to classify data temperature as hot,warm,and cold.With assigned data temperature,the proposed algorithm adaptively selects the appropriate data center and places data accordingly to overcome the excess latency from the WAN environment,leading to overall system performance degradation.This paper also conducts simulations to evaluate the proposed and existing distributed storage methods.The result shows that our proposed method reduced latency by 38%compared to the existing method.Therefore,the proposed method in this paper can be used in large-scale distributed storage systems over a WAN environment to improve latency and performance compared to existing methods,such as consistent hashing.
基金This work was supported by the National Natural Science Foundation of China under Grant 62072249,61772280,61772454,62072056.J.Wang and Y.Ren received the grants,and the URL of the sponsors’website is http://www.nsfc.gov.cn/This work was also supported by the Project of Transformation and Upgrading of Industries and Information Technologies of Jiangsu Province(No.JITC-1900AX2038/01).X.Yu received the grant,and the URL of the sponsors’website is http://gxt.jiangsu.gov.cn/.
文摘Distributed storage can store data in multiple devices or servers to improve data security.However,in today’s explosive growth of network data,traditional distributed storage scheme is faced with some severe challenges such as insufficient performance,data tampering,and data lose.A distributed storage scheme based on blockchain has been proposed to improve security and efficiency of traditional distributed storage.Under this scheme,the following improvements have been made in this paper.This paper first analyzes the problems faced by distributed storage.Then proposed to build a new distributed storage blockchain scheme with sharding blockchain.The proposed scheme realizes the partitioning of the network and nodes by means of blockchain sharding technology,which can improve the efficiency of data verification between nodes.In addition,this paper uses polynomial commitment to construct a new verifiable secret share scheme called PolyVSS.This new scheme is one of the foundations for building our improved distributed storage blockchain scheme.Compared with the previous scheme,our new scheme does not require a trusted third party and has some new features such as homomorphic and batch opening.The security of VSS can be further improved.Experimental comparisons show that the proposed scheme significantly reduces storage and communication costs.
基金the National Natural Science Foundation of China(No.62172327)。
文摘To ensure the reliability and availability of data,redundancy strategies are always required for distributed storage systems.Erasure coding,one of the representative redundancy strategies,has the advantage of low storage overhead,which facilitates its employment in distributed storage systems.Among the various erasure coding schemes,XOR-based erasure codes are becoming popular due to their high computing speed.When a single-node failure occurs in such coding schemes,a process called data recovery takes place to retrieve the failed node’s lost data from surviving nodes.However,data transmission during the data recovery process usually requires a considerable amount of time.Current research has focused mainly on reducing the amount of data needed for data recovery to reduce the time required for data transmission,but it has encountered problems such as significant complexity and local optima.In this paper,we propose a random search recovery algorithm,named SA-RSR,to speed up single-node failure recovery of XOR-based erasure codes.SA-RSR uses a simulated annealing technique to search for an optimal recovery solution that reads and transmits a minimum amount of data.In addition,this search process can be done in polynomial time.We evaluate SA-RSR with a variety of XOR-based erasure codes in simulations and in a real storage system,Ceph.Experimental results in Ceph show that SA-RSR reduces the amount of data required for recovery by up to 30.0%and improves the performance of data recovery by up to 20.36%compared to the conventional recovery method.
基金National Keystone R&D Program of China(No.2017YFB0803204)Shenzhen Research Programs(JCYJ20170306092030521)+3 种基金the PCL Future Regional Network Facilities for Largescale Experiments and Applications(LZC0019)ZTE University Funding,Natural Science Foundation of China(NSFC)(No.61671001)GuangDong Prov.,R&D Key Program(No.2019B010137001)the Shenzhen Municipal Development and Reform Commission(Disciplinary Development Program for Data Science and Intelligent Computing).
文摘With the advent of the era of big data,cloud computing,Internet of things,and other information industries continue to develop.There is an increasing amount of unstructured data such as pictures,audio,and video on the Internet.And the distributed object storage system has become the mainstream cloud storage solution.With the increasing number of distributed applications,data security in the distributed object storage system has become the focus.For the distributed object storage system,traditional defenses are means that fix discovered system vulnerabilities and backdoors by patching,or means to modify the corresponding structure and upgrade.However,these two kinds of means are hysteretic and hardly deal with unknown security threats.Based on mimic defense theory,this paper constructs the principle framework of the distributed object storage system and introduces the dynamic redundancy and heterogeneous function in the distributed object storage system architecture,which increases the attack cost,and greatly improves the security and availability of data.
基金supported by the State Grid Corporation of China Science and Technological Project(Research and demonstration application of key technology of energy storage cloud for mobile energy storage application of electric vehicles 5419-20197121 7a-0-0-00)
文摘Based on the energy storage cloud platform architecture,this study considers the extensive configuration of energy storage devices and the future large-scale application of electric vehicles at the customer side to build a new mode of smart power consumption with a flexible interaction,smooth the peak/valley difference of the load side power,and improve energy efficiency.A plug and play device for customer-side energy storage and an internet-based energy storage cloud platform are developed herein to build a new intelligent power consumption mode with a flexible interaction suitable for ordinary customers.Based on the load perception of the power grid,this study aims to investigate the operating state and service life of distributed energy storage devices.By selecting an integrated optimal control scheme,this study designs a kind of energy optimization and deployment strategy for stratified partition to reduce the operating cost of the energy storage device on the client side.The effectiveness of the system and the control strategy is verified through the Suzhou client-side distributed energy storage demonstration project.
基金supported in part by the National Natural Science Foundation of China (61640006, 61572188)the Natural Science Foundation of Shaanxi Province, China (2015JM6307, 2016JQ6011)the project of science and technology of Xi’an City (2017088CG/RC051(CADX002))
文摘In distributed cloud storage systems, inevitably there exist multiple node failures at the same time. The existing methods of regenerating codes, including minimum storage regenerating(MSR) codes and minimum bandwidth regenerating(MBR) codes, are mainly to repair one single or several failed nodes, unable to meet the repair need of distributed cloud storage systems. In this paper, we present locally minimum storage regenerating(LMSR) codes to recover multiple failed nodes at the same time. Specifically, the nodes in distributed cloud storage systems are divided into multiple local groups, and in each local group(4, 2) or(5, 3) MSR codes are constructed. Moreover, the grouping method of storage nodes and the repairing process of failed nodes in local groups are studied. Theoretical analysis shows that LMSR codes can achieve the same storage overhead as MSR codes. Furthermore, we verify by means of simulation that, compared with MSR codes, LMSR codes can reduce the repair bandwidth and disk I/O overhead effectively.
基金supported by State Grid Corporation Limited Science and Technology Project Funding(Contract No.SGCQSQ00YJJS2200380).
文摘There is instability in the distributed energy storage cloud group end region on the power grid side.In order to avoid large-scale fluctuating charging and discharging in the power grid environment and make the capacitor components showa continuous and stable charging and discharging state,a hierarchical time-sharing configuration algorithm of distributed energy storage cloud group end region on the power grid side based on multi-scale and multi feature convolution neural network is proposed.Firstly,a voltage stability analysis model based onmulti-scale and multi feature convolution neural network is constructed,and the multi-scale and multi feature convolution neural network is optimized based on Self-OrganizingMaps(SOM)algorithm to analyze the voltage stability of the cloud group end region of distributed energy storage on the grid side under the framework of credibility.According to the optimal scheduling objectives and network size,the distributed robust optimal configuration control model is solved under the framework of coordinated optimal scheduling at multiple time scales;Finally,the time series characteristics of regional power grid load and distributed generation are analyzed.According to the regional hierarchical time-sharing configuration model of“cloud”,“group”and“end”layer,the grid side distributed energy storage cloud group end regional hierarchical time-sharing configuration algorithm is realized.The experimental results show that after applying this algorithm,the best grid side distributed energy storage configuration scheme can be determined,and the stability of grid side distributed energy storage cloud group end region layered timesharing configuration can be improved.
基金This work is supported by the Fundamental Research Funds for the Central Universities(Grant No.HIT.NSRIF.201714)Weihai Science and Technology Development Program(2016DX GJMS15)+1 种基金Weihai Scientific Research and Innovation Fund(2020)Key Research and Development Program in Shandong Provincial(2017GGX90103).
文摘The knowledge graph with relational abundant information has been widely used as the basic data support for the retrieval platforms.Image and text descriptions added to the knowledge graph enrich the node information,which accounts for the advantage of the multi-modal knowledge graph.In the field of cross-modal retrieval platforms,multi-modal knowledge graphs can help to improve retrieval accuracy and efficiency because of the abundant relational infor-mation provided by knowledge graphs.The representation learning method is sig-nificant to the application of multi-modal knowledge graphs.This paper proposes a distributed collaborative vector retrieval platform(DCRL-KG)using the multi-modal knowledge graph VisualSem as the foundation to achieve efficient and high-precision multimodal data retrieval.Firstly,use distributed technology to classify and store the data in the knowledge graph to improve retrieval efficiency.Secondly,this paper uses BabelNet to expand the knowledge graph through multi-ple filtering processes and increase the diversification of information.Finally,this paper builds a variety of retrieval models to achieve the fusion of retrieval results through linear combination methods to achieve high-precision language retrieval and image retrieval.The paper uses sentence retrieval and image retrieval experi-ments to prove that the platform can optimize the storage structure of the multi-modal knowledge graph and have good performance in multi-modal space.
基金This work is supported by‘The Fundamental Research Funds for the Central Universities(Grant No.HIT.NSRIF.201714)’‘Weihai Science and Technology Development Program(2016DXGJMS15)’‘Key Research and Development Program in Shandong Provincial(2017GGX90103)’.
文摘In distributed storage systems,file access efficiency has an important impact on the real-time nature of information forensics.As a popular approach to improve file accessing efficiency,prefetching model can fetches data before it is needed according to the file access pattern,which can reduce the I/O waiting time and increase the system concurrency.However,prefetching model needs to mine the degree of association between files to ensure the accuracy of prefetching.In the massive small file situation,the sheer volume of files poses a challenge to the efficiency and accuracy of relevance mining.In this paper,we propose a massive files prefetching model based on LSTM neural network with cache transaction strategy to improve file access efficiency.Firstly,we propose a file clustering algorithm based on temporal locality and spatial locality to reduce the computational complexity.Secondly,we propose a definition of cache transaction according to files occurrence in cache instead of time-offset distance based methods to extract file block feature accurately.Lastly,we innovatively propose a file access prediction algorithm based on LSTM neural network which predict the file that have high possibility to be accessed.Experiments show that compared with the traditional LRU and the plain grouping methods,the proposed model notably increase the cache hit rate and effectively reduces the I/O wait time.
基金Supported by the National Natural Science Foundation of China(61271174,61301178)
文摘We study the construction of minimum bandwidth regenerating code with combinatorial design. At first, a method of constructing minimum storage regenerating （MBR） codes is presented, which can tolerate only one-node failure. Then, we give examples to explain the code. Finally, we discuss the case of repairing multiple nodes, and analyze the performance with an example.
基金This work is supported by the National Key Research and Development Program(No.2022YFB2702101)Shaanxi Key Industrial Province Projects(2021ZDLGY03-02,2021ZDLGY03-08)the National Natural Science Foundation of China under Grants 62272394 and 92152301.
文摘The proliferation of Internet of Things(IoT)systems has resulted in the generation of substantial data,presenting new challenges in reliable storage and trustworthy sharing.Conventional distributed storage systems are hindered by centralized management and lack traceability,while blockchain systems are limited by low capacity and high latency.To address these challenges,the present study investigates the reliable storage and trustworthy sharing of IoT data,and presents a novel system architecture that integrates on-chain and off-chain data manage systems.This architecture,integrating blockchain and distributed storage technologies,provides high-capacity,high-performance,traceable,and verifiable data storage and access.The on-chain system,built on Hyperledger Fabric,manages metadata,verification data,and permission information of the raw data.The off-chain system,implemented using IPFS Cluster,ensures the reliable storage and efficient access to massive files.A collaborative storage server is designed to integrate on-chain and off-chain operation interfaces,facilitating comprehensive data operations.We provide a unified access interface for user-friendly system interaction.Extensive testing validates the system’s reliability and stable performance.The proposed approach significantly enhances storage capacity compared to standalone blockchain systems.Rigorous reliability tests consistently yield positive outcomes.With average upload and download throughputs of roughly 20 and 30 MB/s,respectively,the system’s throughput surpasses the blockchain system by a factor of 4 to 18.
基金the National Natural Science Foundation of China(Nos.61136002,61272120,61634004 and 61602377)the Shaanxi Provincial Co-ordination Innovation Project of Science and Technology(No.2016KTZDGY02-04-02)+1 种基金the Shaanxi Provincial Science and Technology Research Fund(Nos.2013KTZB01-07,2014ZS-08 and S2015TQGY0166)the Fund of Shaanxi Education Bureau(No.2050205)
文摘Vehicular Ad hoc Networks(VANETs)become a very crucial addition in the Intelligent Transportation System(ITS).It is challenging for a VANET system to provide security services and parallelly maintain high throughput by utilizing limited resources.To overcome these challenges,we propose a blockchain-based Secured Cluster-based MAC(SCB-MAC)protocol.The nearby vehicles heading towards the same direction will form a cluster and each of the clusters has its blockchain to store and distribute the safety messages.The message which contains emergency information and requires Strict Delay Requirement(SDR)for transmission are called safety messages(SM).Cluster Members(CMs)sign SMs with their private keys while sending them to the blockchain to confirm authentication,integrity,and confidentiality of the message.A Certificate Authority(CA)is responsible for physical verification,key generation,and privacy preservation of the vehicles.We implemented a test scenario as proof of concept and tested the safety message transmission(SMT)protocol in a real-world platform.Computational and storage overhead analysis shows that the proposed protocol for SMT implements security,authentication,integrity,robustness,non-repudiation,etc.while maintaining the SDR.Messages that are less important compared to the SMs are called non-safety messages(NSM)and vehicles use RTS/CTS mechanism for NSM transmission.Numerical studies show that the proposed NSM transmission method maintains 6 times more throughput,2 times less delay and 125%less Packet Dropping Rate(PDR)than traditional MAC protocols.These results prove that the proposed protocol outperforms the traditional MAC protocols.
基金Supported by the National Major Science and Technology Project“Safe and efficient development technologies for high-sulphur gas reservoirs(PhaseⅢ)”(No.:2016ZX05017).
文摘During the initial construction of the Puguang Gas Field,information infrastructure was built.Due to the absence of unified planning and deployment,however,many“isolated information islands”were formed in data systems,and the data resources cannot meet the construction requirements of intelligent gas field.In this paper,the status quo and problems of data resources in the Puguang Gas Field were analyzed,and a data resource sharing center was constructed according to the overall architecture design of the Puguang Intelligent Gas Field.Based on the architecture design of data resource sharing center,the overall construction conception of data resource sharing center was put forward and the business data model was designed.Finally,the integrated data collection,storage,calculation and utilization was realized by establishing data standard,combing data sources and designing data services,and then it was applied on site.The following research results were obtained.First,the data resource sharing center is an important foundation for the construction of this project,and its overall architecture is divided into three layers from bottom to top,i.e.,data specification and standard,data collection,storage,calculation and utilization,and data control.Second,the data resource sharing center achieves the one-time collection,centralized storage,shared use and unified management of exploration&development,gathering&purification,production&operation and safety&environmental protection data,and provides an important data base for the construction of business system of the intelligent gas field and a comprehensive,reliable and effective data support for the intelligent and mobile application in the Puguang Gas Field.
