Heart failure prediction is crucial as cardiovascular diseases become the leading cause of death worldwide,exacerbated by the COVID-19 pandemic.Age,cholesterol,and blood pressure datasets are becoming inadequate becau...Heart failure prediction is crucial as cardiovascular diseases become the leading cause of death worldwide,exacerbated by the COVID-19 pandemic.Age,cholesterol,and blood pressure datasets are becoming inadequate because they cannot capture the complexity of emerging health indicators.These high-dimensional and heterogeneous datasets make traditional machine learning methods difficult,and Skewness and other new biomarkers and psychosocial factors bias the model’s heart health prediction across diverse patient profiles.Modern medical datasets’complexity and high dimensionality challenge traditional predictionmodels like SupportVectorMachines and Decision Trees.Quantum approaches include QSVM,QkNN,QDT,and others.These Constraints drove research.The“QHF-CS:Quantum-Enhanced Heart Failure Prediction using Quantum CNN with Optimized Feature Qubit Selection with Cuckoo Search in Skewed Clinical Data”system was developed in this research.This novel system leverages a Quantum Convolutional Neural Network(QCNN)-based quantum circuit,enhanced by meta-heuristic algorithms—Cuckoo SearchOptimization(CSO),Artificial BeeColony(ABC),and Particle SwarmOptimization(PSO)—for feature qubit selection.Among these,CSO demonstrated superior performance by consistently identifying the most optimal and least skewed feature subsets,which were then encoded into quantum states for circuit construction.By integrating advanced quantum circuit feature maps like ZZFeatureMap,RealAmplitudes,and EfficientSU2,the QHF-CS model efficiently processes complex,high-dimensional data,capturing intricate patterns that classical models overlook.The QHF-CS model improves precision,recall,F1-score,and accuracy to 0.94,0.95,0.94,and 0.94.Quantum computing could revolutionize heart failure diagnostics by improving model accuracy and computational efficiency,enabling complex healthcare diagnostic breakthroughs.展开更多
The analysis of spatially correlated binary data observed on lattices is an interesting topic that catches the attention of many scholars of different scientific fields like epidemiology, medicine, agriculture, biolog...The analysis of spatially correlated binary data observed on lattices is an interesting topic that catches the attention of many scholars of different scientific fields like epidemiology, medicine, agriculture, biology, geology and geography. To overcome the encountered difficulties upon fitting the autologistic regression model to analyze such data via Bayesian and/or Markov chain Monte Carlo (MCMC) techniques, the Gaussian latent variable model has been enrolled in the methodology. Assuming a normal distribution for the latent random variable may not be realistic and wrong, normal assumptions might cause bias in parameter estimates and affect the accuracy of results and inferences. Thus, it entails more flexible prior distributions for the latent variable in the spatial models. A review of the recent literature in spatial statistics shows that there is an increasing tendency in presenting models that are involving skew distributions, especially skew-normal ones. In this study, a skew-normal latent variable modeling was developed in Bayesian analysis of the spatially correlated binary data that were acquired on uncorrelated lattices. The proposed methodology was applied in inspecting spatial dependency and related factors of tooth caries occurrences in a sample of students of Yasuj University of Medical Sciences, Yasuj, Iran. The results indicated that the skew-normal latent variable model had validity and it made a decent criterion that fitted caries data.展开更多
MapReduce is a widely used programming model for large-scale data processing.However,it still suffers from the skew problem,which refers to the case in which load is imbalanced among tasks.This problem can cause a sma...MapReduce is a widely used programming model for large-scale data processing.However,it still suffers from the skew problem,which refers to the case in which load is imbalanced among tasks.This problem can cause a small number of tasks to consume much more time than other tasks,thereby prolonging the total job completion time.Existing solutions to this problem commonly predict the loads of tasks and then rebalance the load among them.However,solutions of this kind often incur high performance overhead due to the load prediction and rebalancing.Moreover,existing solutions target the partitioning skew for reduce tasks,but cannot mitigate the computational skew for map tasks.Accordingly,in this paper,we present DynamicAdjust,a run-time dynamic resource adjustment technique for mitigating skew.Rather than rebalancing the load among tasks,DynamicAdjust monitors the run-time execution of tasks and dynamically increases resources for those tasks that require more computation.In so doing,DynamicAdjust can not only eliminate the overhead incurred by load prediction and rebalancing,but also culls both the partitioning skew and the computational skew.Experiments are conducted based on a 21-node real cluster using real-world datasets.The results show that DynamicAdjust can mitigate the negative impact of the skew and shorten the job completion time by up to 40.85%.展开更多
This paper focuses on the parallel aggregation processing of data streams based on the shared-nothing architecture. A novel granularity-aware parallel aggregating model is proposed. It employs parallel sampling and li...This paper focuses on the parallel aggregation processing of data streams based on the shared-nothing architecture. A novel granularity-aware parallel aggregating model is proposed. It employs parallel sampling and linear regression to describe the characteristics of the data quantity in the query window in order to determine the partition granularity of tuples, and utilizes equal depth histogram to implement partitio ning. This method can avoid data skew and reduce communi cation cost. The experiment results on both synthetic data and actual data prove that the proposed method is efficient, practical and suitable for time-varying data streams processing.展开更多
Spark作为通用的计算引擎,以其简单、快速、可扩展的优势,被广泛地应用于大数据的处理和分析中.然而,Spark默认采用哈希分区或范围分区对数据进行划分,导致其在处理键倾斜分布的数据时,常常出现各分区数据量严重不均衡的问题.诸多优化...Spark作为通用的计算引擎,以其简单、快速、可扩展的优势,被广泛地应用于大数据的处理和分析中.然而,Spark默认采用哈希分区或范围分区对数据进行划分,导致其在处理键倾斜分布的数据时,常常出现各分区数据量严重不均衡的问题.诸多优化方法被提出,如迁移分区、贪心分区、反馈分区等,但往往存在数据传输量大、额外计算成本高、运行时间长等问题.为更好地缓解键倾斜分布问题带来的影响,本文提出了一种自适应的Spark数据均衡分区方法.该方法引入了奖惩思想对数据分区过程进行适当调控,同时对于数据量较大的键进行分割,使得各个分区的数据量相对均衡.该方法首先对数据采样并预估键权重.其次,按照键权重对样本数据降序排列,确保所有分区都有初始数据.再次,根据奖惩分配策略,自适应地更新各个分区的分配概率,并将待分配的键指向分配概率最高的分区.对于超过分区容量的键的数据,则分割为多个部分且指向不同分区.在所有样本数据分配完成后,获得自适应分区方案.在实际分区时,对于样本中出现的键对应的数据按照自适应分区方案进行分配;对于未出现的键对应的数据,则按照哈希方法进行分区.最后,通过实验验证,基于新方法设计的自适应均衡分区器(Adaptive Data Balanced Partitioner,ADBP)能够有效缓解键倾斜的负面影响.在真实数据集上,ADBP的WordCount程序总运行时间比自带分区器Hash、Range分别平均缩短了1.51%、29.90%,比现有基于学习自动机的自适应哈希分区器(Learning Automata Hash Partitioner,LAHP)、对倾斜的中间数据块进行拆分合并(Splitting and Combination algorithm for skew Intermediate Data block,SCID)算法、粗粒度放置和细粒度放置(Fined-Coarse Grained Intermediate Data Placement,FCGIDP)算法分别平均缩短了8.12%、21.64%、19.62%.展开更多
在证券行业,用户数据处理和分析是核心技术,对业务决策和风险控制具有重要的影响。然而,证券公司庞大的用户数据规模和复杂的数据关系导致大数据计算面临Shuffle操作和数据倾斜问题。现有的Shuffle和数据倾斜优化方法或依赖于硬件升级,...在证券行业,用户数据处理和分析是核心技术,对业务决策和风险控制具有重要的影响。然而,证券公司庞大的用户数据规模和复杂的数据关系导致大数据计算面临Shuffle操作和数据倾斜问题。现有的Shuffle和数据倾斜优化方法或依赖于硬件升级,或存在领域局限性,难以针对性解决该问题。为此,基于证券行业用户数据的特点,提出了一种基于用户关系的多分组归并算法(multi group merging algorithm,MGMA)。该算法通过有效分组和优化处理策略,显著提升计算效率,并降低计算资源消耗。实验表明,相较于无优化对照组,MGMA算法的数据倾斜率为20%,内存占用为72%,计算用时为61%,且上述3项指标均优于其他4种对比优化方法。展开更多
文摘Heart failure prediction is crucial as cardiovascular diseases become the leading cause of death worldwide,exacerbated by the COVID-19 pandemic.Age,cholesterol,and blood pressure datasets are becoming inadequate because they cannot capture the complexity of emerging health indicators.These high-dimensional and heterogeneous datasets make traditional machine learning methods difficult,and Skewness and other new biomarkers and psychosocial factors bias the model’s heart health prediction across diverse patient profiles.Modern medical datasets’complexity and high dimensionality challenge traditional predictionmodels like SupportVectorMachines and Decision Trees.Quantum approaches include QSVM,QkNN,QDT,and others.These Constraints drove research.The“QHF-CS:Quantum-Enhanced Heart Failure Prediction using Quantum CNN with Optimized Feature Qubit Selection with Cuckoo Search in Skewed Clinical Data”system was developed in this research.This novel system leverages a Quantum Convolutional Neural Network(QCNN)-based quantum circuit,enhanced by meta-heuristic algorithms—Cuckoo SearchOptimization(CSO),Artificial BeeColony(ABC),and Particle SwarmOptimization(PSO)—for feature qubit selection.Among these,CSO demonstrated superior performance by consistently identifying the most optimal and least skewed feature subsets,which were then encoded into quantum states for circuit construction.By integrating advanced quantum circuit feature maps like ZZFeatureMap,RealAmplitudes,and EfficientSU2,the QHF-CS model efficiently processes complex,high-dimensional data,capturing intricate patterns that classical models overlook.The QHF-CS model improves precision,recall,F1-score,and accuracy to 0.94,0.95,0.94,and 0.94.Quantum computing could revolutionize heart failure diagnostics by improving model accuracy and computational efficiency,enabling complex healthcare diagnostic breakthroughs.
文摘The analysis of spatially correlated binary data observed on lattices is an interesting topic that catches the attention of many scholars of different scientific fields like epidemiology, medicine, agriculture, biology, geology and geography. To overcome the encountered difficulties upon fitting the autologistic regression model to analyze such data via Bayesian and/or Markov chain Monte Carlo (MCMC) techniques, the Gaussian latent variable model has been enrolled in the methodology. Assuming a normal distribution for the latent random variable may not be realistic and wrong, normal assumptions might cause bias in parameter estimates and affect the accuracy of results and inferences. Thus, it entails more flexible prior distributions for the latent variable in the spatial models. A review of the recent literature in spatial statistics shows that there is an increasing tendency in presenting models that are involving skew distributions, especially skew-normal ones. In this study, a skew-normal latent variable modeling was developed in Bayesian analysis of the spatially correlated binary data that were acquired on uncorrelated lattices. The proposed methodology was applied in inspecting spatial dependency and related factors of tooth caries occurrences in a sample of students of Yasuj University of Medical Sciences, Yasuj, Iran. The results indicated that the skew-normal latent variable model had validity and it made a decent criterion that fitted caries data.
基金funded by the Key Area Research and Development Program of Guangdong Province(2019B010137005)the National Natural Science Foundation of China(61906209).
文摘MapReduce is a widely used programming model for large-scale data processing.However,it still suffers from the skew problem,which refers to the case in which load is imbalanced among tasks.This problem can cause a small number of tasks to consume much more time than other tasks,thereby prolonging the total job completion time.Existing solutions to this problem commonly predict the loads of tasks and then rebalance the load among them.However,solutions of this kind often incur high performance overhead due to the load prediction and rebalancing.Moreover,existing solutions target the partitioning skew for reduce tasks,but cannot mitigate the computational skew for map tasks.Accordingly,in this paper,we present DynamicAdjust,a run-time dynamic resource adjustment technique for mitigating skew.Rather than rebalancing the load among tasks,DynamicAdjust monitors the run-time execution of tasks and dynamically increases resources for those tasks that require more computation.In so doing,DynamicAdjust can not only eliminate the overhead incurred by load prediction and rebalancing,but also culls both the partitioning skew and the computational skew.Experiments are conducted based on a 21-node real cluster using real-world datasets.The results show that DynamicAdjust can mitigate the negative impact of the skew and shorten the job completion time by up to 40.85%.
基金Supported by Foundation of High Technology Pro-ject of Jiangsu (BG2004034) , Foundation of Graduate Creative Pro-gramof Jiangsu (xm04-36)
文摘This paper focuses on the parallel aggregation processing of data streams based on the shared-nothing architecture. A novel granularity-aware parallel aggregating model is proposed. It employs parallel sampling and linear regression to describe the characteristics of the data quantity in the query window in order to determine the partition granularity of tuples, and utilizes equal depth histogram to implement partitio ning. This method can avoid data skew and reduce communi cation cost. The experiment results on both synthetic data and actual data prove that the proposed method is efficient, practical and suitable for time-varying data streams processing.
文摘Spark作为通用的计算引擎,以其简单、快速、可扩展的优势,被广泛地应用于大数据的处理和分析中.然而,Spark默认采用哈希分区或范围分区对数据进行划分,导致其在处理键倾斜分布的数据时,常常出现各分区数据量严重不均衡的问题.诸多优化方法被提出,如迁移分区、贪心分区、反馈分区等,但往往存在数据传输量大、额外计算成本高、运行时间长等问题.为更好地缓解键倾斜分布问题带来的影响,本文提出了一种自适应的Spark数据均衡分区方法.该方法引入了奖惩思想对数据分区过程进行适当调控,同时对于数据量较大的键进行分割,使得各个分区的数据量相对均衡.该方法首先对数据采样并预估键权重.其次,按照键权重对样本数据降序排列,确保所有分区都有初始数据.再次,根据奖惩分配策略,自适应地更新各个分区的分配概率,并将待分配的键指向分配概率最高的分区.对于超过分区容量的键的数据,则分割为多个部分且指向不同分区.在所有样本数据分配完成后,获得自适应分区方案.在实际分区时,对于样本中出现的键对应的数据按照自适应分区方案进行分配;对于未出现的键对应的数据,则按照哈希方法进行分区.最后,通过实验验证,基于新方法设计的自适应均衡分区器(Adaptive Data Balanced Partitioner,ADBP)能够有效缓解键倾斜的负面影响.在真实数据集上,ADBP的WordCount程序总运行时间比自带分区器Hash、Range分别平均缩短了1.51%、29.90%,比现有基于学习自动机的自适应哈希分区器(Learning Automata Hash Partitioner,LAHP)、对倾斜的中间数据块进行拆分合并(Splitting and Combination algorithm for skew Intermediate Data block,SCID)算法、粗粒度放置和细粒度放置(Fined-Coarse Grained Intermediate Data Placement,FCGIDP)算法分别平均缩短了8.12%、21.64%、19.62%.
文摘在证券行业,用户数据处理和分析是核心技术,对业务决策和风险控制具有重要的影响。然而,证券公司庞大的用户数据规模和复杂的数据关系导致大数据计算面临Shuffle操作和数据倾斜问题。现有的Shuffle和数据倾斜优化方法或依赖于硬件升级,或存在领域局限性,难以针对性解决该问题。为此,基于证券行业用户数据的特点,提出了一种基于用户关系的多分组归并算法(multi group merging algorithm,MGMA)。该算法通过有效分组和优化处理策略,显著提升计算效率,并降低计算资源消耗。实验表明,相较于无优化对照组,MGMA算法的数据倾斜率为20%,内存占用为72%,计算用时为61%,且上述3项指标均优于其他4种对比优化方法。
