摘要
MapReduce任务负载均衡主要是通过分区函数来实现的,Hadoop默认的分区函数并不能很好地保证redu-cer的负载均衡。针对周期性的业务处理提出了一种基于权重计算的负载均衡策略,周期性任务的数据分布与历史数据相比具有相似性。本策略根据历史数据运行的信息运算出数据权重信息(文中用权重表示每条记录的处理复杂度),再通过Map阶段抽样分析当前这批数据的分布特征来预测待处理数据带权重的整体近似分布情况,从而指导Reduce分区,以保证其负载均衡。通过简单的例子仿真了整个策略的运作过程,并且对比了与TeraSort思路的不同点。最后通过分析用户访问视频的日志证明了文中提到的策略比默认的策略性能提高了接近1倍。
The MapReduce task load balancing in Hadoop mainly depends on the partition function. The Hadoop default partition function is not efficient in practical business processing. This paper presented a load balancing strategy based on the weight value of the periodic jobs. Because the data＇ s distribution is similar in each period, we calculated the weight from historical data＇s profile. Through analyzing a sample data in Map phase to predict the whole data weighted integral approximate distribution, the strategy guids the Reduce partition to ensure its load balancing. We also presented the difference between TeraSort strategy and the new strategy. The experimental results with the view video logs show that the performance of our strategy is improved about 2 times compared with the default strategy.
出处
《计算机科学》
CSCD
北大核心
2013年第3期38-40,共3页
Computer Science
基金
国家自然科学基金((61272087
61073008
60773148
60503039)
北京市基金(4082016
4122039)资助
