Apparatus, system, method and computer-implemented storage media to implement radio resource management policies using machine learning
An apparatus of a transmitter computing node n (TX node n) of a wireless network, one or more computer readable media, a system, and a method. The apparatus includes one or more processors to: implement machine learning (ML) based training rounds, each training round including: determining a local action value function Q n (h n , a n ; θ n ) corresponding to a value of performing a radio resource management (RRM) action a n at a receiving computing node n (RX node n) associated with TX node n using policy parameter θ n and based on h n , h n including channel state information at RX node n; and determining, based on an overall action value function Q tot at time t, an estimated gradient of an overall loss at time t for overall policy parameter θ t (∇L t (θ t )), wherein Q tot corresponds to a mixing of local action value functions Q i (h i , a i ; θ i ) for all TX nodes i in the network at time t including TX node n; and determine, in response to a determination that ∇L t (θ t ) is close to zero for various values of t during training, a trained local action value function Q n,trained to generate a trained action value relating to data communication between TX node n and RX node n.
1. An apparatus of a transmitter computing node n (TX node n) of a wireless network, the apparatus including an interconnect interface to connect the apparatus to one or more components of the TX node n, and one or more processors to: implement machine learning (ML) based training rounds, each training round including:
determining, using reinforcement learning, a local action value function Q n (h n , a n ; θ n ) corresponding to a value of performing a radio resource management (RRM) action an at a receiving computing node n (RX node n) associated with TX node n using policy parameter θ n and based on h n , h n including channel state information at RX node n; and
determining, based on an overall action value function Q tot at time t, an estimated gradient of an overall loss at time t for overall policy parameter θ t (∇L t (θ t )) wherein Q tot corresponds to a mixing of local action value functions Q i (h i , a i ; θ i ) for all TX nodes i in the network at time t including TX node n; and
determine, in response to a determination that ∇L t (θ t ) is close to zero for various values of t during training, a trained local action value function Q n,trained to generate a trained action value relating to data communication between TX node n and RX node n.
2. The apparatus of claim 1 , the one or more processors to further receive, during said each training round, h n from RX node n, wherein h n includes information on a state of a channel n between the TX node and the RX node n and information h mn on states of respective channels between other TX nodes m and the RX node n.
3. The apparatus of claim 1 , wherein the one or more processors are to determine, during said each training round, an estimated gradient of a local loss at time t for policy parameter θ n,t; (∇L t (θ n,t )) prior to determining the trained local action value function.
4. The apparatus of claim 1 , wherein the one or more processors are to send data for transmission to the RX node n based on the trained action value.
5. The apparatus of claim 1 , wherein the one or more processors are to, at a start of the training rounds, implement an initialization function to initialize a value for a n and θ n .
6. The apparatus of claim 1 , wherein the one or more processors are to, at a start of the training rounds, implement an initialization function to initialize a value for θ, and, at each training round:
determine Q tot , wherein Q tot is based on H, a and θ, wherein H corresponds to a matrix of channel state information between pairs of TX nodes and RX nodes of the network, a corresponds to a vector of RRM decisions for each TX node i of the network, and θ is a policy parameter to map H to a; and
send for transmission Q tot to other TX nodes m of the network.
7. The apparatus of claim 1 , wherein the one or more processors are to send Q n and h n for transmission to a central server, and to receive Q tot from the central server, Q tot corresponding to a mixing of local action value functions Q i (h i , a i ; θ i ) for all TX nodes i in the network at time t including TX node n.
8. The apparatus of claim 1 , wherein determining ∇L t (θ t ) includes using one of a gradient descent operation or a chain rule operation.
9. The apparatus of claim 1 , wherein L t (θ t ) is based on subtracting, from a target value y tot for the overall action value function Qtot, a square of Qtot at time t.
10. The apparatus of claim 1 , further comprising a transceiver coupled to the one or more processors, and one or more antennas coupled to the transceiver, the antennas to send and receive wireless communications in the wireless network.
11. The apparatus of claim 10 , further comprising a system memory coupled to the one or more processors, the system memory to store instructions, the one or more processors to execute the instructions to perform the training.
12. The apparatus of claim 10 , further comprising:
a network interface card (NIC) coupled to the one or more processors to connect the one or more processors to a core network by way of wired access; and
a housing that encloses the one or more processors, the transceiver, and the NIC.
13. A method to be performed at an apparatus of a transmitter computing node n (TX node n) of a wireless network, the method including:
implementing machine learning (ML) based training rounds, each training round including:
determining, using reinforcement learning, a local action value function Q n (h n , a n ; θ n ) corresponding to a value of performing a radio resource management (RRM) action a n at a receiving computing node n (RX node n) associated with TX node n using policy parameter θ n and based on h n , h n including channel state information at RX node n; and
determining, based on an overall action value function Q tot at time t, an estimated gradient of an overall loss at time t for overall policy parameter θ t (∇L t (θ t )) wherein Q tot corresponds to a mixing of local action value functions Q i (h i , a i ; θ i ) for all TX nodes i in the network at time t including TX node n; and
determining, in response to a determination that ∇L t (θ t ) is close to zero for various values of t during training, a trained local action value function Q n,trained to generate a trained action value relating to data communication between TX node n and RX node n.
14. The method of claim 13 , further including receiving, during said each training round, h n from RX node n, wherein h n includes information on a state of a channel n between the TX node and the RX node n and information h mn on states of respective channels between other TX nodes m and the RX node n.
15. The method of claim 13 , further including determining, during said each training round, an estimated gradient of a local loss at time t for policy parameter θ n,t; (∇L t (θ n,t )) prior to determining the trained local action value function.
16. The method of claim 13 , further including sending data for transmission to the RX node n based on the trained action value.
17. The method of claim 13 , further including, at each training round:
determining Q tot , wherein Q tot is based on H, a and θ, wherein H corresponds to a matrix of channel state information between pairs of TX nodes and RX nodes of the network, a corresponds to a vector of RRM decisions for each TX node i of the network, and θ is a policy parameter to map H to a; and
sending for transmission Q tot to other TX nodes m of the network.
18. A non-transitory machine readable storage medium having instructions stored thereon, the instructions when executed by an apparatus of a transmitter computing node n (TX node n) of a wireless network to cause the apparatus to perform operations including:
implementing machine learning (ML) based training rounds, each training round including:
determining, using reinforcement learning, a local action value function Q n (h n , a n ; θ n ) corresponding to a value of performing a radio resource management (RRM) action a n at a receiving computing node n (RX node n) associated with TX node n using policy parameter θ n and based on h n , h n including channel state information at RX node n; and
determining, based on an overall action value function Q tot at time t, an estimated gradient of an overall loss at time t for overall policy parameter θ t (∇L t (θ t )) wherein Qtot corresponds to a mixing of local action value functions Q i (h i , a i ; θ i ) for all TX nodes i in the network at time t including TX node n; and
determine, in response to a determination that ∇L t (θ t ) is close to zero for various values of t during training, a trained local action value function Q n,trained to generate a trained action value relating to data communication between TX node n and RX node n.
19. The storage medium of claim 18 , the operations further including receiving, during said each training round, h n from RX node n, wherein h n includes information on a state of a channel n between the TX node and the RX node n and information h mn on states of respective channels between other TX nodes m and the RX node n.
20. The storage medium of claim 18 , wherein Q is discounted long term value.
21. The storage medium of claim 18 , the operations further including determining, during said each training round, an estimated gradient of a local loss at time t for policy parameter θ n,t; (∇L t (θ n,t )) prior to determining the trained local action value function.
22. The storage medium of claim 18 , the operations further including sending data for transmission to the RX node n based on the trained action value.
23. The storage medium of claim 18 , the operations further including, before the training rounds, implementing an initialization function to initialize a value for a n and θ n .
24. The storage medium of claim 18 , the operations further including, at each training round:
determining Q tot , wherein Q tot is based on H, a and θ, wherein H corresponds to a matrix of channel state information between pairs of TX nodes and RX nodes of the network, a corresponds to a vector of RRM decisions for each TX node i of the network, and θ is a policy parameter to map H to a; and
sending for transmission Q tot to other TX nodes m of the network.
25. The storage medium of claim 18 , the operations further including sending Q n and h n for transmission to a central server, and to receive Q tot from the central server, Q tot corresponding to a mixing of local action value functions Q i (h i , a i ; θ i ) for all TX nodes i in the network at time t including TX node n.