Data quality has exerted important influence over the application of grain big data, so data cleaning is a necessary and important work. In MapReduce frame, parallel technique is often used to execute data cleaning in...Data quality has exerted important influence over the application of grain big data, so data cleaning is a necessary and important work. In MapReduce frame, parallel technique is often used to execute data cleaning in high scalability mode, but due to the lack of effective design, there are amounts of computing redundancy in the process of data cleaning, which results in lower performance. In this research, we found that some tasks often are carried out multiple times on same input files, or require same operation results in the process of data cleaning. For this problem, we proposed a new optimization technique that is based on task merge. By merging simple or redundancy computations on same input files, the number of the loop computation in MapReduce can be reduced greatly. The experiment shows, by this means, the overall system runtime is significantly reduced, which proves that the process of data cleaning is optimized. In this paper, we optimized several modules of data cleaning such as entity identification, inconsistent data restoration, and missing value filling. Experimental results show that the proposed method in this paper can increase efficiency for grain big data cleaning.展开更多
Task duplication has been widely adopted to mitigate the impact of stragglers that run much longer than normal tasks. However,task duplication on data pipelining case would generate excessive traffic over the datacent...Task duplication has been widely adopted to mitigate the impact of stragglers that run much longer than normal tasks. However,task duplication on data pipelining case would generate excessive traffic over the datacenter networks. In this paper, we study minimizing the traffic cost for data pipelining task replications and design a controller that chooses the data generated by the first finished task and discards data generated later by other replications belonging to the same task. Each task replication communicates with the controller when it finishes a data processing, which causes additional network overhead. Hence, we try to reduce the network overhead and make a trade-off between the delay of data block and the network overhead. Finally, extensive simulation results demonstrate that our proposal can minimize network traffic cost under data pipelining case.展开更多
以旅游大数据为基础,考虑长时间范围内的滞后效应以及不同搜索强度指数(Search Intensity Index,SII)之间的多任务影响,提出一种基于大数据的多任务旅游信息分析(Multi-tasking Tourism Information Analysis Based on Big Data,MTIABD...以旅游大数据为基础,考虑长时间范围内的滞后效应以及不同搜索强度指数(Search Intensity Index,SII)之间的多任务影响,提出一种基于大数据的多任务旅游信息分析(Multi-tasking Tourism Information Analysis Based on Big Data,MTIABD)框架。使用融合信息重排序技术预测旅游需求,具体根据图引导结构模拟历史变量对未来变量的滞后影响。每个变量通过时间维度上的卷积神经网络(Convolutional Neural Network,CNN)进行独立编码,利用二分图动态建模滞后效应,通过图聚合进行挖掘,实现对旅游需求的精准预测。基于上述技术,构建旅游需求预测系统,旅游者能够根据需求检索不同景点的信息。在真实数据集上进行大量实验,结果表明所提出的MTIABD框架在一步和多步预测方面均优于现有方法。在平均绝对百分比误差(Mean Absolute Percentage Error,MAPE)指标下,相较于基于实例的多变量时间序列图预测框架(Instance-wise Graph-rased Framework for Multivariate Time Series Forecasting,IGMTF),MTIABD在HK-2021数据集上的性能提高了16.75%,在MO-2021数据集上的性能提高了19.79%。展开更多
浮动车数据(floating car data, FCD)技术是大规模城市路网交通流实时采集的有效方法.城市交通的动态诱导和控制需要对海量FCD进行快速处理.鉴于此,提出了FCD并行计算的动态任务调度方法.针对FCD数据包计算时间的不确定性和动态性,根据...浮动车数据(floating car data, FCD)技术是大规模城市路网交通流实时采集的有效方法.城市交通的动态诱导和控制需要对海量FCD进行快速处理.鉴于此,提出了FCD并行计算的动态任务调度方法.针对FCD数据包计算时间的不确定性和动态性,根据计算节点的处理能力进行数据包的动态分割,在处理过程中,采用动态任务分配策略以实现计算节点的同步.该方法在龙芯国产大数据一体机平台上进行了实现,并采用现场FCD数据进行了实验验证,结果表明,该方法较轮询和Min-Min调度算法,显著地提高了并行处理的性能.展开更多
文摘Data quality has exerted important influence over the application of grain big data, so data cleaning is a necessary and important work. In MapReduce frame, parallel technique is often used to execute data cleaning in high scalability mode, but due to the lack of effective design, there are amounts of computing redundancy in the process of data cleaning, which results in lower performance. In this research, we found that some tasks often are carried out multiple times on same input files, or require same operation results in the process of data cleaning. For this problem, we proposed a new optimization technique that is based on task merge. By merging simple or redundancy computations on same input files, the number of the loop computation in MapReduce can be reduced greatly. The experiment shows, by this means, the overall system runtime is significantly reduced, which proves that the process of data cleaning is optimized. In this paper, we optimized several modules of data cleaning such as entity identification, inconsistent data restoration, and missing value filling. Experimental results show that the proposed method in this paper can increase efficiency for grain big data cleaning.
文摘Task duplication has been widely adopted to mitigate the impact of stragglers that run much longer than normal tasks. However,task duplication on data pipelining case would generate excessive traffic over the datacenter networks. In this paper, we study minimizing the traffic cost for data pipelining task replications and design a controller that chooses the data generated by the first finished task and discards data generated later by other replications belonging to the same task. Each task replication communicates with the controller when it finishes a data processing, which causes additional network overhead. Hence, we try to reduce the network overhead and make a trade-off between the delay of data block and the network overhead. Finally, extensive simulation results demonstrate that our proposal can minimize network traffic cost under data pipelining case.
文摘以旅游大数据为基础,考虑长时间范围内的滞后效应以及不同搜索强度指数(Search Intensity Index,SII)之间的多任务影响,提出一种基于大数据的多任务旅游信息分析(Multi-tasking Tourism Information Analysis Based on Big Data,MTIABD)框架。使用融合信息重排序技术预测旅游需求,具体根据图引导结构模拟历史变量对未来变量的滞后影响。每个变量通过时间维度上的卷积神经网络(Convolutional Neural Network,CNN)进行独立编码,利用二分图动态建模滞后效应,通过图聚合进行挖掘,实现对旅游需求的精准预测。基于上述技术,构建旅游需求预测系统,旅游者能够根据需求检索不同景点的信息。在真实数据集上进行大量实验,结果表明所提出的MTIABD框架在一步和多步预测方面均优于现有方法。在平均绝对百分比误差(Mean Absolute Percentage Error,MAPE)指标下,相较于基于实例的多变量时间序列图预测框架(Instance-wise Graph-rased Framework for Multivariate Time Series Forecasting,IGMTF),MTIABD在HK-2021数据集上的性能提高了16.75%,在MO-2021数据集上的性能提高了19.79%。
文摘浮动车数据(floating car data, FCD)技术是大规模城市路网交通流实时采集的有效方法.城市交通的动态诱导和控制需要对海量FCD进行快速处理.鉴于此,提出了FCD并行计算的动态任务调度方法.针对FCD数据包计算时间的不确定性和动态性,根据计算节点的处理能力进行数据包的动态分割,在处理过程中,采用动态任务分配策略以实现计算节点的同步.该方法在龙芯国产大数据一体机平台上进行了实现,并采用现场FCD数据进行了实验验证,结果表明,该方法较轮询和Min-Min调度算法,显著地提高了并行处理的性能.
