In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of ...In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.展开更多
In this paper, a novel approach termed process goose queue (PGQ) is suggested to deal with real-time optimization (RTO) of chemical plants. Taking advantage of the ad-hoc structure of PGQ which imitates biologic natur...In this paper, a novel approach termed process goose queue (PGQ) is suggested to deal with real-time optimization (RTO) of chemical plants. Taking advantage of the ad-hoc structure of PGQ which imitates biologic nature of flying wild geese, a chemical plant optimization problem can be re-formulated as a combination of a multi-layer PGQ and a PGQ-Objective according to the relationship among process variables involved in the objective and constraints. Subsequently, chemical plant RTO solutions are converted into coordination issues among PGQs which could be dealt with in a novel way. Accordingly, theoretical definitions, adjustment rule and implementing procedures associated with the approach are explicitly introduced together with corresponding enabling algorithms. Finally, an exemplary chemical plant is employed to demonstrate the feasibility and validity of the contribution.展开更多
Without considering security, existing message scheduling mechanisms may expose critical messages to malicious threats like confidentiality attacks. Incorporating confidentiality improvement into message scheduling, t...Without considering security, existing message scheduling mechanisms may expose critical messages to malicious threats like confidentiality attacks. Incorporating confidentiality improvement into message scheduling, this paper investigates the problem of scheduling aperiodc messages with time-critical and security-critical requirements. A risk-based security profit model is built to quantify the security quality of messages; and a dynamic programming based approximation algorithm is proposed to schedule aperiodic messages with guaranteed security performance. Experimental results illustrate the efficiency and effectiveness of the proposed algorithm.展开更多
Nowadays, we experience an abundance of Internet of Things middleware solutions that make the sensors and the actuators are able to connect to the Internet. These solutions, referred to as platforms to gain a widespre...Nowadays, we experience an abundance of Internet of Things middleware solutions that make the sensors and the actuators are able to connect to the Internet. These solutions, referred to as platforms to gain a widespread adoption, have to meet the expectations of different players in the IoT ecosystem, including devices [1]. Low cost devices are easily able to connect wirelessly to the Internet, from handhelds to coffee machines, also known as Internet of Things (IoT). This research describes the methodology and the development process of creating an IoT platform. This paper also presents the architecture and implementation for the IoT platform. The goal of this research is to develop an analytics engine which can gather sensor data from different devices and provide the ability to gain meaningful information from IoT data and act on it using machine learning algorithms. The proposed system is introducing the use of a messaging system to improve the overall system performance as well as provide easy scalability.展开更多
One of the most important features of modem minor satellites is to realize autonomous moving. The perfomance of the satellite autonomous computer operating system acting as the control center is of utrnost importance....One of the most important features of modem minor satellites is to realize autonomous moving. The perfomance of the satellite autonomous computer operating system acting as the control center is of utrnost importance. The recent trend in operating system development is adopting microkernel architecture that holds such advantages as microminiaturization, modularity, portability and extendibility. IPC is the key of microkernel design. Message-based IPC mechanism is generally used in existing microkernel Operating system. It is of consistency, safety and reliability.However, it can not provide efficient support for real-time applications in satellite systems and it only applies to loose coupling multi-processor architecture. In this paper, an improvement solution for existing message-based IPC is proposed at first to obtain real-time performance. Then a new IPC mechanism is designed. It particulary applies to shared memory tight coupling multi-processor architecture.展开更多
By thorough research on the prominent periodic and aperiodic scheduling algorithms,anon-line hard real-time scheduler is presented,which is applicable to the scheduling of packets over a link.This scheduler,based on b...By thorough research on the prominent periodic and aperiodic scheduling algorithms,anon-line hard real-time scheduler is presented,which is applicable to the scheduling of packets over a link.This scheduler,based on both Rate Monotonic,pinwheel scheduling algorithm Sr and Polling Serverscheduling algorithms,can rapidly judge the schedulability and then automatically generate a bus tablefor the scheduling algorithm to schedule the packets as the periodic packets.The implementation of thescheduler is simple and easy to use,and it is effective for the utilization of bus link.The orderly executionof the bus table can not only guarantee the performance of the hard real time but also avoid the blockageand interruption of the message transmission.So the scheduler perfectly meets the demand of hard real-time communication system on the field bus domain.展开更多
We consider a single server constant retrial queue,in which a state-dependent service policy is used to control the service rate.Customer arrival follows Poisson process,while service time and retrial time are exponen...We consider a single server constant retrial queue,in which a state-dependent service policy is used to control the service rate.Customer arrival follows Poisson process,while service time and retrial time are exponential distributions.Whenever the server is available,it admits the retrial customers into service based on a first-come first-served rule.The service rate adjusts in real-time based on the retrial queue length.An iterative algorithm is proposed to numerically solve the personal optimal problem in the fully observable scenario.Furthermore,we investigate the impact of parameters on the social optimal threshold.The effectiveness of the results is illustrated by two examples.展开更多
Time-sensitive networks(TSNs)support not only traditional best-effort communications but also deterministic communications,which send each packet at a deterministic time so that the data transmissions of networked con...Time-sensitive networks(TSNs)support not only traditional best-effort communications but also deterministic communications,which send each packet at a deterministic time so that the data transmissions of networked control systems can be precisely scheduled to guarantee hard real-time constraints.No-wait scheduling is suitable for such TSNs and generates the schedules of deterministic communications with the minimal network resources so that all of the remaining resources can be used to improve the throughput of best-effort communications.However,due to inappropriate message fragmentation,the realtime performance of no-wait scheduling algorithms is reduced.Therefore,in this paper,joint algorithms of message fragmentation and no-wait scheduling are proposed.First,a specification for the joint problem based on optimization modulo theories is proposed so that off-the-shelf solvers can be used to find optimal solutions.Second,to improve the scalability of our algorithm,the worst-case delay of messages is analyzed,and then,based on the analysis,a heuristic algorithm is proposed to construct low-delay schedules.Finally,we conduct extensive test cases to evaluate our proposed algorithms.The evaluation results indicate that,compared to existing algorithms,the proposed joint algorithm improves schedulability by up to 50%.展开更多
With the increasing data volume of train on-board system,real-time performance has become the most critical factor to ensure the safety of train operation.Considering that standard Ethernet cannot meet the real-time r...With the increasing data volume of train on-board system,real-time performance has become the most critical factor to ensure the safety of train operation.Considering that standard Ethernet cannot meet the real-time requirement of existing train communication network(TCN),the time-sensitive network(TSN)technology for TCN is introduced.To solve the time-delay problem,an adaptive switch queue selection mechanism for traffic scheduling is proposed.Firstly,the topology model of TCN based on TSN and the traffic model are described.Then,the K shortest path routing algorithm based on load balancing provides the optimal routing for the scheduling process.Finally,the adaptive switch queue selection mechanism is introduced to solve the aggregation flow conflict problem effectively,queue resources are properly allocated,and the gate control list(GCL)of each frame in the queue is obtained.Experimental results show that compared with the traditional constraint model,the schedulability of the model with an adaptive switch queue selection mechanism increases by 33.0%,and the maximum end-to-end delay and network jitter decrease by 19.1%and 18.6%on average respectively.It can provide theoretical support and application reference for the real-time performance optimization of TCN based on TSN.展开更多
基金supported by the Research Fund of National Key Laboratory of Computer Architecture under Grant No.CARCH201501the Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing under Grant No.2016A09
文摘In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.
文摘In this paper, a novel approach termed process goose queue (PGQ) is suggested to deal with real-time optimization (RTO) of chemical plants. Taking advantage of the ad-hoc structure of PGQ which imitates biologic nature of flying wild geese, a chemical plant optimization problem can be re-formulated as a combination of a multi-layer PGQ and a PGQ-Objective according to the relationship among process variables involved in the objective and constraints. Subsequently, chemical plant RTO solutions are converted into coordination issues among PGQs which could be dealt with in a novel way. Accordingly, theoretical definitions, adjustment rule and implementing procedures associated with the approach are explicitly introduced together with corresponding enabling algorithms. Finally, an exemplary chemical plant is employed to demonstrate the feasibility and validity of the contribution.
基金supported by the National Natural Science Foundation of China (60673142)the National High Technology Research and Development Progrm of China (863 Program) (2006AA01Z1732007AA01Z131)
文摘Without considering security, existing message scheduling mechanisms may expose critical messages to malicious threats like confidentiality attacks. Incorporating confidentiality improvement into message scheduling, this paper investigates the problem of scheduling aperiodc messages with time-critical and security-critical requirements. A risk-based security profit model is built to quantify the security quality of messages; and a dynamic programming based approximation algorithm is proposed to schedule aperiodic messages with guaranteed security performance. Experimental results illustrate the efficiency and effectiveness of the proposed algorithm.
文摘Nowadays, we experience an abundance of Internet of Things middleware solutions that make the sensors and the actuators are able to connect to the Internet. These solutions, referred to as platforms to gain a widespread adoption, have to meet the expectations of different players in the IoT ecosystem, including devices [1]. Low cost devices are easily able to connect wirelessly to the Internet, from handhelds to coffee machines, also known as Internet of Things (IoT). This research describes the methodology and the development process of creating an IoT platform. This paper also presents the architecture and implementation for the IoT platform. The goal of this research is to develop an analytics engine which can gather sensor data from different devices and provide the ability to gain meaningful information from IoT data and act on it using machine learning algorithms. The proposed system is introducing the use of a messaging system to improve the overall system performance as well as provide easy scalability.
文摘One of the most important features of modem minor satellites is to realize autonomous moving. The perfomance of the satellite autonomous computer operating system acting as the control center is of utrnost importance. The recent trend in operating system development is adopting microkernel architecture that holds such advantages as microminiaturization, modularity, portability and extendibility. IPC is the key of microkernel design. Message-based IPC mechanism is generally used in existing microkernel Operating system. It is of consistency, safety and reliability.However, it can not provide efficient support for real-time applications in satellite systems and it only applies to loose coupling multi-processor architecture. In this paper, an improvement solution for existing message-based IPC is proposed at first to obtain real-time performance. Then a new IPC mechanism is designed. It particulary applies to shared memory tight coupling multi-processor architecture.
基金Supported by the Emphases Science and Technology Project Foundation of Sichuan Province(NO.02GG006-037)
文摘By thorough research on the prominent periodic and aperiodic scheduling algorithms,anon-line hard real-time scheduler is presented,which is applicable to the scheduling of packets over a link.This scheduler,based on both Rate Monotonic,pinwheel scheduling algorithm Sr and Polling Serverscheduling algorithms,can rapidly judge the schedulability and then automatically generate a bus tablefor the scheduling algorithm to schedule the packets as the periodic packets.The implementation of thescheduler is simple and easy to use,and it is effective for the utilization of bus link.The orderly executionof the bus table can not only guarantee the performance of the hard real time but also avoid the blockageand interruption of the message transmission.So the scheduler perfectly meets the demand of hard real-time communication system on the field bus domain.
基金supported by the National Natural Science Foundation of China(Grant No.11971486)。
文摘We consider a single server constant retrial queue,in which a state-dependent service policy is used to control the service rate.Customer arrival follows Poisson process,while service time and retrial time are exponential distributions.Whenever the server is available,it admits the retrial customers into service based on a first-come first-served rule.The service rate adjusts in real-time based on the retrial queue length.An iterative algorithm is proposed to numerically solve the personal optimal problem in the fully observable scenario.Furthermore,we investigate the impact of parameters on the social optimal threshold.The effectiveness of the results is illustrated by two examples.
基金partially supported by National Key Research and Development Program of China(2018YFB1700200)National Natural Science Foundation of China(61972389,61903356,61803368,U1908212)+2 种基金Youth Innovation Promotion Association of the Chinese Academy of Sciences,National Science and Technology Major Project(2017ZX02101007-004)Liaoning Provincial Natural Science Foundation of China(2020-MS-034,2019-YQ-09)China Postdoctoral Science Foundation(2019M661156)。
文摘Time-sensitive networks(TSNs)support not only traditional best-effort communications but also deterministic communications,which send each packet at a deterministic time so that the data transmissions of networked control systems can be precisely scheduled to guarantee hard real-time constraints.No-wait scheduling is suitable for such TSNs and generates the schedules of deterministic communications with the minimal network resources so that all of the remaining resources can be used to improve the throughput of best-effort communications.However,due to inappropriate message fragmentation,the realtime performance of no-wait scheduling algorithms is reduced.Therefore,in this paper,joint algorithms of message fragmentation and no-wait scheduling are proposed.First,a specification for the joint problem based on optimization modulo theories is proposed so that off-the-shelf solvers can be used to find optimal solutions.Second,to improve the scalability of our algorithm,the worst-case delay of messages is analyzed,and then,based on the analysis,a heuristic algorithm is proposed to construct low-delay schedules.Finally,we conduct extensive test cases to evaluate our proposed algorithms.The evaluation results indicate that,compared to existing algorithms,the proposed joint algorithm improves schedulability by up to 50%.
文摘消息队列作为一种常见的数据结构,在应用程序和软件操作系统中得到了广泛应用,而随着处理器的不断迭代,传统的锁同步消息队列逐渐无法发挥多核处理器的性能。因此,研究人员尝试引入布谷鸟过滤器(Cuckoo Filter,CCF)和DQueue技术对消息队列进行优化升级,创建云开发消息队列(Cloud Development Message Queue,CDMQ)。基于上述两种技术同时生成多个队列写入数据,操作者利用CCF对数据进行过滤检验,并读取相关信息。通过对CDMQ的性能测试,研究人员证明该消息队列结构具有伸缩性强和吞吐量高等优势,对提高消息队列性能具有重要意义。
基金supported by the National Natural Science Foundation of China(52072081)Major Project of Science and Technology of Guangxi Province of China(Guike AB23075209)+2 种基金Guangxi Manufacturing Systems and Advanced Manufacturing Technology Key Laboratory Director Fund(24050-44-S015)Innovation Project of Guangxi Graduate Education(YCSW2024135)Major Talent Project in Guangxi Zhuang Autonomous Region。
文摘With the increasing data volume of train on-board system,real-time performance has become the most critical factor to ensure the safety of train operation.Considering that standard Ethernet cannot meet the real-time requirement of existing train communication network(TCN),the time-sensitive network(TSN)technology for TCN is introduced.To solve the time-delay problem,an adaptive switch queue selection mechanism for traffic scheduling is proposed.Firstly,the topology model of TCN based on TSN and the traffic model are described.Then,the K shortest path routing algorithm based on load balancing provides the optimal routing for the scheduling process.Finally,the adaptive switch queue selection mechanism is introduced to solve the aggregation flow conflict problem effectively,queue resources are properly allocated,and the gate control list(GCL)of each frame in the queue is obtained.Experimental results show that compared with the traditional constraint model,the schedulability of the model with an adaptive switch queue selection mechanism increases by 33.0%,and the maximum end-to-end delay and network jitter decrease by 19.1%and 18.6%on average respectively.It can provide theoretical support and application reference for the real-time performance optimization of TCN based on TSN.
