Fresh status updates are vital to the efficient operation of network monitoring and real-time control applications. In this paper, we consider a mobile edge computing(MEC)-assisted status update system, where smart de...Fresh status updates are vital to the efficient operation of network monitoring and real-time control applications. In this paper, we consider a mobile edge computing(MEC)-assisted status update system, where smart devices extract valuable status updates from sensed data to achieve timely awareness of the surroundings by exploiting computational resources at the device and edge server. To quantify the freshness of status updates obtained by executing computation tasks, we employ the concept of age of information(Ao I) to characterize the timeliness of status updates. To cope with the limited energy at devices, we investigate a joint task generation and computation offloading scheme under a given energy budget for minimizing the age of obtained status updates. The age minimization problem is modeled as a constrained Markov decision process(CMDP). To obtain the optimal policy, we derive the structural properties of the optimal deterministic policy and propose a lightweight structure-based status update algorithm in the case of known channel statistics. Moreover, we consider a more realistic scenario without prior knowledge of channel statistics, and propose a Q-learning-based status update algorithm to make online decisions. Simulation results show that the performance of our proposed algorithms is competitive when compared with existing schemes.展开更多
1 Introduction Constrained Reinforcement Learning(CRL),modeled as a Constrained Markov Decision Process(CMDP)[1,2],is commonly used to address applications with security restrictions.Previous works[3]primarily focused...1 Introduction Constrained Reinforcement Learning(CRL),modeled as a Constrained Markov Decision Process(CMDP)[1,2],is commonly used to address applications with security restrictions.Previous works[3]primarily focused on the single-constraint issue,overlooking the more common multi-constraint setting which involves extensive computations and combinatorial optimization of multiple Lagrange multipliers.展开更多
This paper studies a dynamic multi-user wireless network,where users have no knowledge of the arrival rate and size of data block and suffer from a constraint on long-term average power consumption.Considering such a ...This paper studies a dynamic multi-user wireless network,where users have no knowledge of the arrival rate and size of data block and suffer from a constraint on long-term average power consumption.Considering such a network,we address the problem of dynamically optimizing channel/power allocation,so as to minimize the long-term average data backlog.The design problem is shown to be a constrained Markov decision process.In order to solve the problem without knowledge on dynamics of the system,we introduce post-decision states and propose a resource allocation algorithm based on reinforcement learning.Since the channel/power allocation problem is coupled,the multiuser decision problem suffers from curses of dimensions(of state/action/outcome space).This makes centralized decision-making and optimization on channel/power allocation suffer from a long convergence time.As a countermeasure,a partially distributed resource allocation framework is proposed.The multiuser power allocation problem is decoupled into single-user decision problems,while channel allocation optimization is performed in a centralized manner.In order to further reduce computational complexity,we propose a low-complexity reinforcement learning method.Simulation results reveal that the proposed algorithm outperforms the state-of-the-art myopic optimizations in terms of energy efficiency and the backlog performance.展开更多
基金supported in part by National Science Foundation for Young Scientists of China Project No.042700349Beijing Natural Science Foundation under Grant 19L2033Key Area R&D Program of Guangdong Province with grant No.2018B030338001。
文摘Fresh status updates are vital to the efficient operation of network monitoring and real-time control applications. In this paper, we consider a mobile edge computing(MEC)-assisted status update system, where smart devices extract valuable status updates from sensed data to achieve timely awareness of the surroundings by exploiting computational resources at the device and edge server. To quantify the freshness of status updates obtained by executing computation tasks, we employ the concept of age of information(Ao I) to characterize the timeliness of status updates. To cope with the limited energy at devices, we investigate a joint task generation and computation offloading scheme under a given energy budget for minimizing the age of obtained status updates. The age minimization problem is modeled as a constrained Markov decision process(CMDP). To obtain the optimal policy, we derive the structural properties of the optimal deterministic policy and propose a lightweight structure-based status update algorithm in the case of known channel statistics. Moreover, we consider a more realistic scenario without prior knowledge of channel statistics, and propose a Q-learning-based status update algorithm to make online decisions. Simulation results show that the performance of our proposed algorithms is competitive when compared with existing schemes.
基金supported by the Fundamental Research Funds for the Central Universities(No.2023JBZX011)the Aeronautical Science Foundation of China(No.202300010M5001).
文摘1 Introduction Constrained Reinforcement Learning(CRL),modeled as a Constrained Markov Decision Process(CMDP)[1,2],is commonly used to address applications with security restrictions.Previous works[3]primarily focused on the single-constraint issue,overlooking the more common multi-constraint setting which involves extensive computations and combinatorial optimization of multiple Lagrange multipliers.
基金This work was supported in part by National Natural Science Foundation of China under Grant 61901216,61631020 and 61827801Natural Science Foundation of Jiangsu Province under Grant BK20190400+1 种基金Open Research Fund of National Mobile Communications Research Laboratory,Southeast University(No.2020D08)Foundation of Graduate Innovation Center in NUAA under Grant kfjj20190408。
文摘This paper studies a dynamic multi-user wireless network,where users have no knowledge of the arrival rate and size of data block and suffer from a constraint on long-term average power consumption.Considering such a network,we address the problem of dynamically optimizing channel/power allocation,so as to minimize the long-term average data backlog.The design problem is shown to be a constrained Markov decision process.In order to solve the problem without knowledge on dynamics of the system,we introduce post-decision states and propose a resource allocation algorithm based on reinforcement learning.Since the channel/power allocation problem is coupled,the multiuser decision problem suffers from curses of dimensions(of state/action/outcome space).This makes centralized decision-making and optimization on channel/power allocation suffer from a long convergence time.As a countermeasure,a partially distributed resource allocation framework is proposed.The multiuser power allocation problem is decoupled into single-user decision problems,while channel allocation optimization is performed in a centralized manner.In order to further reduce computational complexity,we propose a low-complexity reinforcement learning method.Simulation results reveal that the proposed algorithm outperforms the state-of-the-art myopic optimizations in terms of energy efficiency and the backlog performance.
