Graph neural network and reinforcement learning techniques for connection management
The present disclosure provides connection management techniques based on graph neural networks (GNN) and deep reinforcement learning (DRL) to optimize user association and load balancing. A graph structure of a communication network is considered for the GNN architecture and DRL is used to learn parameters of the GNN algorithm/model. Connection management is defined as a combinatorial graph optimization problem, and the DRL mechanism uses the underlying graph to learn weights of the GNN for an optimal user connections or associations. The connection management techniques can consider local network features to make better decisions to balance network traffic load while network throughput is also maximized. Implementations are provided based on edge computing frameworks include the Open RAN (O-RAN) architecture. Other embodiments may be described and/or claimed.
1. An apparatus comprising:
memory circuitry;
computer-readable instructions;
at least one programmable circuit to be programmed based on the computer-readable instructions to:
identify, after detection of a connection event, communication equipment instances (CEs) of a set of CEs and communication links of a set of communication links, the communication links between at least pairs of CEs in the set of CEs;
generate an initial graph including an initial arrangement of edges among a set of nodes, the set of nodes representative of respective CEs of the set of CEs, the edges in the initial arrangement of edges representative of respective communication links of the set of communication links;
cause a graph neural network (GNN) to determine a set of candidate graphs based on a set of node features, respective ones of the candidate graphs including corresponding candidate arrangements of edges between corresponding ones of the set of nodes, the candidate arrangements different than the initial arrangement;
determine respective quality (Q) values of the candidate graphs based on a Q function, the Q function based on a reinforcement learning model;
select at least one of the candidate graphs from among the set of candidate graphs based on the Q values, the selected at least one of the candidate graphs including an output arrangement of edges between corresponding ones of the nodes; and
generate connection management (CM) instructions to reconfigure the set of communication links between one or more CEs of the set of CEs based on the output arrangement of edges; and
interface circuitry to send the CM instructions to the one or more CEs in the set of CEs, the CM instructions to cause the one or more CEs to establish one or more communication links with other CEs in the set of CEs.
2. The apparatus of claim 1 , wherein a first Q value for a first candidate graph in the set of candidate graphs is an expected reward value associated with rearrangement of the set of communication links according to a first one of the candidate arrangements corresponding to the first candidate graph.
3. The apparatus of claim 2 , wherein the GNN includes a plurality of GNN layers, the plurality of GNN layers including at least an input layer, an output layer, and at least one hidden layer disposed between the input layer and the output layer, and the output layer of the GNN is an input to the Q function.
4. The apparatus of claim 1 , wherein, to determine the set of candidate graphs, one or more of the at least one programmable circuit is to:
identify, based on the initial graph, a current state of a communication network including the set of CEs; and
for a first candidate graph in the set of candidate graphs:
determine respective Q values for the edges in a first one of the candidate arrangements, the first one of the candidate arrangements corresponding to the first candidate graph; and
combine the respective Q values to obtain the Q value of the first candidate graph.
5. The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to:
operate the GNN based on at least one of throughput of a communication network including the set of CEs, coverage of the communication network, or load balance among the set of CEs.
6. The apparatus of claim 1 , wherein the set of node features includes data rates of the set of CEs and channel capacities of the set of CEs, and wherein:
the interface circuitry is to obtain network metrics from individual CEs of the set of CEs; and
one or more of the at least one programmable circuit is to determine the channel capacities and the data rates based on the obtained network metrics.
7. The apparatus of claim 6 , wherein the network metrics include one or more measurement reports, wherein a measurement report of the one or more measurement reports includes at least one signal or channel measurement, a CE identifier of a CE that performed the at least one signal or channel measurement, and an identifier of a coverage area in which the CE performed the at least one signal or channel measurement.
8. The apparatus of claim 7 , wherein the network metrics include at least one of respective CE rates of the individual CEs, respective CE spectral efficiency metrics of the individual CEs, respective CE resource utilization of the individual CEs, a total resource utilization associated with the set of CEs, respective bandwidth utilization at the individual CEs, a total bandwidth utilization associated with the set of CEs, or CE status information that indicated whether ones of the individual CEs are in an active mode, an inactive mode, or an idle mode.
9. The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to:
cause storage of connection events in a connection event queue; and
process the queued connection events in a first in first out manner.
10. The apparatus of claim 9 , wherein one or more of the at least one programmable circuit is to process a number of the queued connection events in parallel.
11. The apparatus of claim 1 , wherein the set of CEs includes a set of network access nodes (NANs) and a set of user equipment (UEs), individual NANs of the set of NANs provide network connectivity to one or more UEs of the set of UEs, and the communication links in the set of communication links are between ones of the UEs and ones of the NANs.
12. The apparatus of claim 11 , wherein one or more of the at least one programmable circuit is to generate, as the initial graph, a NAN-NAN adjacency graph based on locations of the individual NANs in the set of NANs.
13. The apparatus of claim 12 , wherein to one or more of the at least one programmable circuit is to generate the NAN-NAN adjacency graph to include a first NAN in the set of NANs serving a UE that is a subject of the detected connection event and a subset of second NANs in the set of NANs within a predetermined distance from the first NAN.
14. The apparatus of claim 13 , wherein one or more of the at least one programmable circuit is to generate the NAN-NAN adjacency graph based on a score calculated based on network metrics collected from individual UEs of the set of UEs.
15. The apparatus of claim 11 , wherein the apparatus corresponds to a Radio Access Network (RAN) Intelligent Controller (RIC) in an Open RAN (O-RAN) framework, the computer-readable instructions correspond to an xApp operated by the RIC, and the individual NANs are at least one of O-RAN distributed units (DUs) or O-RAN remote units (RUs).
16. At least one computer-readable medium comprising instructions to cause at least one programmable circuit to at least:
identify, after detection of a connection event, communication equipment instances (CEs) of a set of CEs and communication links of a set of communication links, the communication links between at least pairs of CEs in the set of CEs;
generate an initial graph including an initial arrangement of edges among a set of nodes, the set of nodes representative of respective CEs of the set of CEs, the edges in the initial arrangement of edges representative of respective communication links of the set of communication links;
cause a graph neural network (GNN) to determine a set of candidate graphs based on a set of node features, respective ones of the candidate graphs including corresponding candidate arrangements of edges between corresponding ones of the set of nodes, the candidate arrangements different than the initial arrangement;
determine respective quality (Q) values of the candidate graphs based on a reinforcement learning model;
select at least one of the candidate graphs from among the set of candidate graphs based on the Q values, the selected at least one of the candidate graphs including an output arrangement of edges between corresponding ones of the nodes;
generate connection management (CM) instructions to reconfigure the set of communication links between one or more CEs of the set of CEs based on the output arrangement of edges; and
send the CM instructions to the one or more CEs in the set of CEs, the CM instructions to cause the one or more CEs to establish one or more communication links with other CEs in the set of CEs.
17. The at least one non-transitory computer-readable medium of claim 16 , wherein a Q function, a first Q value for a first candidate graph in the set of candidate graphs is an expected reward value associated with rearrangement of the set of communication links according to a first one of the candidate arrangements corresponding to the first candidate graph.
18. The at least one non-transitory computer-readable medium of claim 17 , wherein the instructions are to cause one or more of the at least one programmable circuit to:
identify, based on the initial graph, a current state of a communication network including the set of CEs; and
for a first candidate graph in the set of candidate graphs:
determine respective Q values for the edges in a first one of the candidate arrangements, the first one of the candidate arrangements corresponding to the first candidate graph; and
combine the respective Q values to obtain the Q value of the first candidate graph.
19. The at least one non-transitory computer-readable medium of claim 18 , wherein the instructions are to cause one or more of the at least one programmable circuit to perform, during a training phase of the GNN, stochastic gradient descent (SGD) to minimize a squared loss function to learn one or more weights of the GNN.
20. The at least one non-transitory computer-readable medium of claim 16 , wherein the instructions are to cause one or more of the at least one programmable circuit to operate the GNN based on at least one of throughput of a communication network including the set of CEs, coverage of the communication network, or load balance among the set of CEs.
21. The at least one non-transitory computer-readable medium of claim 16 , wherein the set of node features includes data rates of the set of CEs and channel capacities of the set of CEs, and the instructions are to cause one or more of the at least one programmable circuit to:
obtain network metrics from individual CEs of the set of CEs; and
determine the channel capacities and the data rates based on the obtained network metrics.
22. The at least one non-transitory computer-readable medium of claim 21 , wherein the network metrics include one or more measurement reports, wherein a measurement report of the one or more measurement reports includes at least one signal or channel measurement, a CE identifier of a CE that performed the at least one signal or channel measurement, and an identifier of a coverage area in which the CE performed the at least one signal or channel measurement.
23. The at least one non-transitory computer-readable medium of claim 22 , wherein the at least one signal or channel measurement includes at least one of a received signal reference power (RSRP) measurement, a reference signal received quality (RSRQ) measurement, a Received Signal Code Power (RSCP) measurement, a signal-to-noise ratio (SNR) measurement, a signal-to-noise and interference ratio (SINR) measurement, a channel quality indicator (CQI) measurement, or an energy per chip to noise power density ratio (EcN0) measurement.
24. The at least one non-transitory computer-readable medium of claim 21 , wherein the network metrics include at least one of respective CE rates of the individual CEs, respective CE spectral efficiency metrics of the individual CEs, respective CE resource utilization of the individual CEs, a total resource utilization associated with the set of CEs, respective bandwidth utilization at the individual CEs, a total bandwidth utilization associated with the set of CEs, or CE status information that indicated whether ones of the individual CEs are in an active mode, an inactive mode, or an idle mode.
25. The at least one non-transitory computer-readable medium of claim 21 , wherein the instructions are to cause one or more of the at least one programmable circuit to determine the channel capacities and the data rates based on historical network metrics.
26. The at least one non-transitory computer-readable medium of claim 21 , wherein the instructions are to cause one or more of the at least one programmable circuit to determine the set of node features for a layer in the GNN such that features of individual nodes of the set of nodes are propagated to one or more neighboring nodes of the set of nodes.
27. The at least one non-transitory computer-readable medium of claim 16 , wherein the detected connection event is one of receipt of a measurement report, receipt of an intra-radio access technology (RAT) handover (HO) request, receipt of an inter-RAT HO request, receipt of a cell selection message, receipt of a cell reselection message, receipt of a radio link failure (RLF) detection message, receipt of an RLF recovery message(s), receipt of a beam failure detection message, or a beam recovery message.