摘要
为解决海量气象数据并发处理的技术难点,提出了气象数据综合权重算法以及Kafka并发处理性能最优策略。选取湖南省2020年6月气象数据作为实验数据集,提取对数据处理系统性能影响最显著的关键特征值,基于熵权法计算关键特征值在流转和处理时消耗基础资源的综合权重,并将其作为气象数据分区处理的客观依据。参照气象大数据云平台架构,设计以Kafka为核心的数据处理模型,通过实验分别得出气象数据在Producer和Consumer端最优Partition、Thread配置策略,从而提升并发处理能力。实验结果表明:对实验数据集进行分区并配置最优策略后,在有限基础资源支撑条件下,消息写入速度从0.69 MB/s提升至37.44 MB/s,消息读取速度从15.65 MB/s提升至67.34 MB/s。该算法和策略已应用在气象卫星遥感数据处理业务,有效解决了海量卫星遥感数据传输处理过程出现消息阻塞的现象,在各类数据处理系统设计中具有较强的参考价值。
To solve the key challenges in the concurrent processing of massive meteorological data,a new comprehensive weight algorithm of meteorological data and the optimal strategy of Kafka concurrent processing performance are developed.The meteorological data of Hunan Province in June 2020 was selected as the experimental data set,and the key eigenvalue that had the most significant influence on the performance of the data processing system were extracted.Based on the entropy weight method,the comprehensive weight of the basic resources consumed by the key eigenvalue during the flow and processing was calculated,which was used as the objective basis for the partition processing of meteorological data.With reference to the meteorological big data cloud platform architecture,a data processing model with Kafka as the core is designed,and the optimal Partition and Thread configuration strategies for meteorological data in the Producer and Consumer are obtained through experiments,so as to improve the concurrent processing capability.The experimental results show that the message writing speed is improved from 0.69 MB/s to 37.44 MB/s,and the reading speed is improved from 15.65 MB/s to 67.34 MB/s after the optimal strategy is configured for the processing of the experimental data set on limited basic resources.The algorithm and strategy have been applied to the meteorological satellite remote sensing data processing business,effectively solving the message blocking phenomenon in the transmission and processing of massive satellite remote sensing data,which have strong reference value in the design of various data processing systems.
作者
冯冼
方昆
文立恒
朱宏武
FENG Xian;FANG Kun;WEN Li-heng;ZHU Hong-wu(Hunan Meteorological Information Center,Changsha 410118,China;Hunan Key Laboratory of Meteorological Disaster Prevention and Reduction,Changsha 410118,China)
出处
《计算机技术与发展》
2023年第8期214-220,共7页
Computer Technology and Development
基金
湖南省自然科学基金资助项目(2020JJ4397)
湖南省气象局重点科研基金资助项目(NLJS2019-07)。
关键词
气象数据
关键特征值
权重算法
并发处理
分区策略
meteorological data
key eigenvalue
weight algorithm
concurrent processing
partition policy
