Reinforcement learning for multi-access traffic management
The present disclosure is related to multi-access traffic management in edge computing environments, and in particular, artificial intelligence (AI) and/or machine learning (ML) techniques for multi-access traffic management. A scalable AI/ML architecture for multi-access traffic management is provided. Reinforcement learning (RL) and/or Deep RL (DRL) approaches that learn policies and/or parameters for traffic management and/or for distributing multi-access traffic through interacting with the environment are also provided. Deep contextual bandit RL techniques for intelligent traffic management for edge networks are also provided. Other embodiments may be described and/or claimed.
1. An apparatus comprising:
interface circuitry to collect observation data from a plurality of multi-access user equipment (UE) within an environment;
a non-transitory machine-readable medium including machine-readable instructions; and
at least one processor circuit to be programmed based on the machine-readable instructions to:
determine a state of the environment based on the collected observation data;
operate a Reinforcement Learning Model (RLM) to determine one or more actions based on the determined state, the one or more actions to cause one or more of the multi-access UEs to perform one or more operations according to one or more traffic management strategies;
obtain action data associated with one or more other actions taken by one or more of the multi-access UEs;
train the RLM based on the observation data and the action data; and
deploy the RLM to the one or more of the multi-access UEs, the deployed RLM to identify one or more subsequent actions to be taken by at least one of the multi-access UEs.
2. The apparatus of claim 1 , wherein the one or more traffic management strategies include one or more of a traffic steering strategy or a traffic splitting strategy.
3. The apparatus of claim 1 , wherein the interface circuitry is to communicate the one or more actions to one or more of the multi-access UEs.
4. The apparatus of claim 1 , wherein one or more of the multi-access UEs are to use the deployed RLM to independently predict one or more of the traffic management strategies according to their individually collected observations from interacting with the environment.
5. The apparatus of claim 4 , wherein one or more of the at least one processor circuit is to:
calculate a reward value based on the collected observation data;
update the RLM based on the calculated reward value; and
distribute the updated RLM to the one or more multi-access UEs when the updated RLM passes a verification process.
6. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to:
trigger collection of the observation data when a trigger condition is met, the trigger condition including one or more of expiration of a timer or a measurement value meeting a threshold.
7. The apparatus of claim 1 , wherein the interface circuitry is to collect additional observation data from one or more network access nodes (NANs).
8. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to:
calculate, based on a reward function, a reward value based on the collected observation data.
9. The apparatus of claim 8 , wherein the reward function is a utility function of a network quality of service (QOS) target.
10. The apparatus of claim 9 , wherein inputs to the utility function include one or more QoS parameters, the one or more QOS parameters including one or more of packet delay, packet loss rate, packet drop rate, physical (PHY) rate, goodput, UE throughput, cell throughput, jitter, alpha fairness, channel quality indicator (CQI) related measurements, Modulation Coding Scheme (MCS) related measurements, physical resource block (PRB) utilization, radio utilization level per NAN, or data volume.
11. The apparatus of claim 1 , wherein the observation data includes one or more of packet delay, packet loss rate, packet drop rate, PHY rate, goodput, UE throughput, cell throughput, jitter, alpha fairness, CQI related measurements, MCS related measurements, PRB utilization, radio utilization level per NAN, or data volume.
12. The apparatus of claim 1 , wherein the apparatus corresponds to a Multi-Access Edge Computing (MEC) server/host or an Open RAN Alliance (O-RAN) Radio Access Network (RAN) intelligent controller (RIC).
13. One or more non-transitory computer readable media (NTCRM) comprising instructions to cause at least one processor circuit of a computing device to at least:
operate a representation network of a Reinforcement Learning Model (RLM) to generate a representation of a multi-access communication environment based on observation data collected from the multi-access communication environment;
operate an actor network of the RLM to determine one or more actions based on the generated representation and feedback from a critic network of the RLM, the one or more actions including one or more traffic management strategies for one or more of a plurality of multi-access user equipment (UE), and the one or more actions are to cause one or more of the multi-access UEs to perform one or more operations according to the one or more traffic management strategies;
operate the critic network to determine the feedback for the one or more actions;
communicate the one or more actions to one or more of the multi-access UEs;
obtain action data associated with one or more other actions taken by one or more of the multi-access UEs;
train the RLM based on the observation data and the action data; and
deploy the RLM to one or more of the multi-access UEs, execution of the deployed RLM to derive subsequent actions by one or more of the multi-access UEs.
14. The one or more NTCRM of claim 13 , wherein the feedback is (i) a state-value based on a state-value function or (ii) a quality value (Q value) based on a Q value function.
15. The one or more NTCRM of claim 13 , wherein the feedback includes a probability distribution over the one or more actions.
16. The one or more NTCRM of claim 13 , wherein the representation network includes a recurrent neural network (RNN) to learn the representation of the multi-access communication environment based on inputs having variable sizes.
17. The one or more NTCRM of claim 13 , wherein the critic network includes an RNN that is to learn a measurement sequence correlation to determine the feedback.
18. The one or more NTCRM of claim 13 , wherein the actor network includes an RNN that is to learn a measurement sequence correlation to determine the one or more actions.
19. The one or more NTCRM of claim 13 , wherein the representation network, the actor network, and the critic network include respective Long Short-Term Memory (LSTM) networks.
20. The one or more NTCRM of claim 13 , wherein the instructions are to cause one or more of the at least one processor circuit of the computing device to:
randomly initialize the critic network, the actor network, and the representation network with respective weight parameters;
initialize a replay buffer;
initialize a random process to perform action exploration; and
operate the representation network to obtain a representation of the observation data for a target UE to whom an action recommendation is to be provided, the target UE is among the multi-access UEs.
21. The one or more NTCRM of claim 20 , wherein the instructions are to cause one or more of the at least one processor circuit of the computing device to:
operate the actor network to select an action according to a current policy and exploration noise, and deploy the selected action to the target UE;
collect a reward based on performance of the selected action; and
cause storage of the selected action, a state, and the collected reward in the replay buffer.
22. The one or more NTCRM of claim 21 , wherein, to select the action, the instructions are to cause one or more of the at least one processor circuit of the computing device to:
randomly sample an action space for the action based on a probability; or
select the action based on a heuristic algorithm.
23. The one or more NTCRM of claim 21 , wherein the instructions are to cause one or more of the at least one processor circuit of the computing device to:
sample a minibatch of experience data from the replay buffer;
determine a temporal difference (TD) target for a TD error computation based on the collected reward; and
train the critic network to minimize a loss function that includes the TD target.
24. The one or more NTCRM of claim 23 , wherein the instructions are to cause one or more of the at least one processor circuit of the computing device to:
train the actor network based on a policy gradient to maximize a reward function, the policy gradient produced by the critic network.
25. An apparatus comprising:
interface circuitry to collect observation data from a multi-access communication environment;
a non-transitory machine-readable medium including machine-readable instructions; and
at least one processor circuit to be programmed based on the machine-readable instructions to:
determine a state of the multi-access communication environment based on the collected observation data; and
operate a Reinforcement Learning (RL) agent to determine one or more actions based on the determined state and an enforce safety action space (ESAS) procedure, the determined one or more actions including one or more traffic management strategies for one or more of a plurality of multi-access user equipment (UE), the one or more actions to cause one or more of the multi-access UEs to perform one or more operations according to the one or more traffic management strategies, and the ESAS procedure based on an acceptable range of actions or a set of constraints described by one or more linear or non-linear functions of the one or more actions.
26. The apparatus of claim 25 , wherein one or more of the at least one processor circuit is to operate the RL agent to determine the one or more actions based on the determined state, the ESAS procedure and a guided exploration procedure, the guided exploration procedure including at least one of one or more rule-based algorithms, one or more model-based heuristic algorithms, or one or more pre-trained machine learning algorithms.
27. The apparatus of claim 25 , wherein one or more of the at least one processor circuit is to operate the RL agent to determine the one or more actions based on the determined state, the ESAS procedure and an early warning procedure, the early warning procedure to:
trigger implementation of a back-up model when one or more performance metrics meet or exceed a corresponding threshold, the one or more performance metrics including one or more of an average value of one way delay for one or more data flows, an average value of end-to-end (e2e) delay for one or more data flows, a delay variation trend, buffer accumulation, a channel quality indicator (CQI) variation trend, or a modulation coding scheme (MCS) distribution trend.
28. The apparatus of claim 25 , wherein one or more of the at least one processor circuit is to operate the RL agent to determine the one or more actions based on the determined state, the ESAS procedure and an opportunistic exploration control (OEC) procedure, the OEC procedure to:
identify one or more of the one or more actions as high risk actions; and
apply the one or more high risk actions to test flows or flows with less stringent QoS targets than other flows.