期刊文献+
任意字段
题名或关键词
题名
关键词
文摘
作者
第一作者
机构
刊名
分类号
参考文献
作者简介
基金资助
栏目信息

一种广域网环境下的分布式冗余删除存储系统 被引量：1

Distributed De-Duplication Storage Systems in Wide Area Networks
在线阅读 下载PDF
导出
摘要 分布式存储系统中存储了海量的数据,这些数据中存在着大量的冗余,将冗余数据删除技术应用到分布式存储系统当中,用来发现并去除数据中的冗余,可以有效地提高存储空间以及网络带宽利用率。文章设计并实现了广域网环境下的分布式冗余删除存储系统——Aegean Store。该系统在数据被上传之前将冗余数据去除,达到提高存储资源和网络资源利用率的目的,并且进一步地降低存储系统的成本,在保持分布式系统固有的容灾特性的同时,可提高存储系统的可扩展性和整体性能。 The problem of redundancy often occurs in mass data stored in distributed storage systems. Greater efficiency in storage and network bandwidth utilization can be achieved by employing de-duplication techniques to eliminate this problem. This article introduces the design and implementation of redundancy-removal systems in a distributed WAN environment. AegeanStore can eliminate redundancy prior to data being uploaded. This frees up storage space and resource use, and lowers the total cost of storage. Furthermore, disaster recovery features inherent in the distributed system are maintained, storage system scalability is enhanced, and overall performance of the network is improved.
作者 尹玉冰 孙竞 余宏亮
机构地区 清华大学高性能计算研究所
出处 《中兴通讯技术》 2010年第5期20-23,共4页 ZTE Technology Journal
基金 国家重点基础研究发展("973")规划(2007CB311100)
关键词 分布式系统 存储系统 冗余数据删除 distributed system storage system de-duplication
分类号 TP333 [自动化与计算机技术—计算机系统结构]
  • 相关文献

参考文献11

  • 1Amazon Simple Storage Service (Amazon S3) [EB/OL]. [2010-06-16]. http://aws.amazon. com/s3/.
  • 2敖莉,舒继武,李明强.重复数据删除技术[J].软件学报,2010,21(5):916-929. 被引量：118
  • 3Cryptographic Hash Function [EB/OL]. {2010-05-25]. http://en.wikipedia.org/wiki/ Cryptographic_hash_functions.
  • 4DENEHY T E, HSU W W. Duplicate Management for Reference Data [R]. Research Report. RJ 10305 (A0310-017). San Jose, CA, USA: IBM. 2003.
  • 5BRODER A Z. identifying and Filtering Near-duplicate Documents [C]ffProceedings of the 1 lth Annual Symposium on Combinatorial Pattern Matching(CPM '00), Jun 21-23, 2000, Montreal, Canada. Berlin, Germany: Sorinaer-Verlag, 2000:1-10.
  • 6HUNTJ W, MCILLROY M C. An Algorithm for Differential File Comparison [R]. Computing Science Technical Report 41. Stanford, CA, USA: Stanford University, 1976.
  • 7TOLIA N, KOZUCH M, SATYANARAYANAN M, et al. Opportunistic Use of Content-addressable Storage for Distributed File Systems [C]//Proceedings of the 2003 USENIX Annual Technical Conference (USENIX'03), Jun 9-14, 2003, San Antonio, TX, USA. Berkeley, CA, USA: USENIX Association. 2003:127-140.
  • 8GUPTA A, LISKOV B, RODRIGUES R. One Hop Lookups for Peer-to-peer Overlays [C]// Proceedings of the 9th Conference on Hot Topics in Operating Systems (HotOS'03), May 18-21, 2003, Lihue, HI,USA. Berkeley, CA, USA: USENIX Association,2003: 7-12.
  • 9BRODER A Z. Some Applications of Rabin's Fingerprinting Method [M]//CAPOCELLI R, DE SANTIS A, VACCARO U. Sequences Ⅱ: Methods in Communications, Security, and Computer Science. Berlin, Germany: Springer-Verlag, 1993:143-152.
  • 10ESH-GHI K, TANG H K. A Framework for Analyzing and Improving Content-based Chunking Algorithms [R]. TR 2005-30. Hewlett-Packard Labs. 2009.

二级参考文献42

  • 1Bhagwat D,Pollack K,Long DDE,Schwarz T,Miller EL,P-ris JF.Providing high reliability in a minimum redundancy archival storage system.In:Proc.of the 14th Int'l Symp.on Modeling,Analysis,and Simulation of Computer and Telecommunication Systems (MASCOTS 2006).Washington:IEEE Computer Society Press,2006.413-421.
  • 2Zhu B,Li K.Avoiding the disk bottleneck in the data domain deduplication file system.In:Proc.of the 6th Usenix Conf.on File and Storage Technologies (FAST 2008).Berkeley:USENIX Association,2008.269-282.
  • 3Bhagwat D,Eshghi K,Mehra P.Content-Based document routing and index partitioning for scalable similarity-based searches in a large corpus.In:Berkhin P,Caruana R,Wu XD,Gaffney S,eds.Proc.of the 13th ACM SIGKDD Int'l Conf.on Knowledge Discovery and Data Mining (KDD 2007).New York:ACM Press,2007.105-112.
  • 4You LL,Pollack KT,Long DDE.Deep store:An archival storage system architecture.In:Proc.of the 21st Int'l Conf.on Data Engineering (ICDE 2005).Washington:IEEE Computer Society Press,2005.804-815.
  • 5Quinlan S,Dorward S.Venti:A new approach to archival storage.In:Proc.of the 1st Usenix Conf.on File and Storage Technologies (FAST 2002).Berkeley:USENIX Association,2002.89-102.
  • 6Sapuntzakis CP,Chandra R,Pfaff B,Chow J,Lam MS,Rosenblum M.Optimizing the migration of virtual computers.In:Proc.of the 5th Symp.on Operating Systems Design and Implementation (OSDI 2002).New York:ACM Press,2002.377-390.
  • 7Rabin MO.Fingerprinting by random polynomials.Technical Report,CRCT TR-15-81,Harvard University,1981.
  • 8Rivest R.The MD5 message-digest algorithm.1992.http://www.python.org/doc/current/lib/module-md5.html.
  • 9U.S.National Institute of Standards and Technology (NIST).Federal Information Processing Standards (FIPS) Publication 180-1:Secure Hash Standard.1995.http://www.itl.nist.gov/fipspubs/fip180-1.htm.
  • 10U.S.National Institute of Standards and Technology (NIST).Federal Information Processing Standards (FIPS) Publication 180-2:Secure Hash Standard.2002.http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf.

共引文献117

同被引文献4

引证文献1

中兴通讯技术
2010年 第5期

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部