Iterative initialization of machine-learning agent parameters in wireless communication network
A machine-learning (ML) orchestrator entity provides distributed, flexible, and efficient parameter initialization and updating for ML agents can be installed on network nodes operating under similar radio conditions. The ML orchestrator entity instructs each of such network nodes to iteratively run the ML agent in a training mode. Each run yields a local set of parameters for the ML agent. After each run, the ML orchestrator entity collects and uses the local sets of parameters from two or more network nodes to derive a common set of parameters for the network nodes. The ML orchestrator further instructs each of the network nodes to update its own local set of parameters based on the common set of parameters and use the updated local set of parameters in a subsequent run. The ML orchestrator entity repeats these steps until a termination criterion for the training mode is met.
1 . A machine-learning (ML) orchestrator entity in a wireless communication network, said ML orchestrator entity comprising:
at least one processor; and
at least one memory including computer program code;
wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the ML orchestrator entity to:
group a set of network nodes in the wireless communication network into at least one node cluster based on at least one radio condition of a set of cells served by the set of network nodes, each network node from the set of network nodes having an ML agent installed thereon, the ML agent being configured to run based on a radio measurement in a training mode and in an inference mode; and
for each node cluster from the at least one node cluster:
transmit, to each network node of the node cluster, a first indication to iteratively run the ML agent in the training mode, each run of the ML agent in the training mode yielding a local set of parameters for the ML agent; and
after each run of the ML agents in the training mode:
receive at least two local sets of parameters from at least two network nodes of the node cluster;
based on the at least two local sets of parameters, obtain a common set of parameters;
based on the common set of parameters, check whether a termination criterion for the training mode is met; and
if the termination criterion for the training mode is not met, transmit, to each network node of the node cluster, a second indication to: (i) update the local set of parameters based on the common set of parameters, and (ii) use the updated local set of parameters in a subsequent run of the ML agent in the training mode, the second indication comprising the common set of parameters.
2 . The ML orchestrator entity of claim 1 , wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the ML orchestrator entity to receive each of the at least two local sets of parameters via an ML agent-specific signalling interface.
3 . The ML orchestrator entity of claim 1 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity to:
after each run of the ML agents in the training mode, transmit a request for the local set of parameters to each of the at least two network nodes of the node cluster; and
in response to the request, receive the at least two local sets of parameters.
4 . The ML orchestrator of claim 3 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity, before transmitting the request, to randomly select the at least two network nodes among the node cluster.
5 . The ML orchestrator entity of claim 1 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity to:
after each run of the ML agents in the training mode, receive a request for the common set of parameters from each network node of the node cluster; and
in response to the request, transmit the second indication to each network node of the node cluster.
6 . The ML orchestrator entity of claim 1 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity to transmit, together with the second indication, a time instant from which the common set of parameters is to be used for updating the local set of parameters at each network node of the node cluster before the subsequent run of the ML agents is initiated.
7 . The ML orchestrator entity of claim 1 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity to:
if the termination criterion for the training mode is met, transmit, to each network node of the node cluster, a third indication to: (i) terminate the training mode for the ML agent, and (ii) run the ML agent in the inference mode by using the common set of parameters, the third indication comprising the common set of parameters.
8 . The ML orchestrator entity of claim 7 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity to transmit, together with the third indication, a time instant from which the common set of parameters is to be used in the inference mode of the ML agent at each network node of the node cluster.
9 . The ML orchestrator entity of claim 1 , wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the ML orchestrator entity to obtain the common set of parameters by using at least one of a linear function, a non-linear function, and a Boolean function.
10 . The ML orchestrator entity of claim 1 , wherein the ML agent comprises a reinforcement learning (RL) agent configured to run in an exploration mode as the training mode and in an exploitation mode as the inference mode.
11 . The ML orchestrator entity of claim 10 , wherein the RL agent is based on a Q-learning approach, and each of the at least two local sets of parameters from the at least two network nodes of the node cluster is presented as a local Q-table, and wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the ML orchestrator entity to obtain the common set of parameters as a common Q-table.
12 . The ML orchestrator entity of claim 1 , wherein the termination criterion for the training mode is defined as:
a similarity metric calculated based on two or more common sets of parameters obtained after two or more consecutive runs of the ML agents of the at least two network nodes in the training mode is less than or equal to a threshold.
13 . A network node in a wireless communication network, said network node comprising:
at least one processor; and
at least one memory including computer program code;
wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the network node to:
receive, from a machine-learning (ML) orchestrator entity, a first indication to iteratively run an ML agent installed on the network node in a training mode, each run of the ML agent in the training mode yielding a local set of parameters for the ML agent; and
after each run of the ML agent in the training mode:
transmit the local set of parameters to the ML orchestrator entity; and
receive, from the ML orchestrator entity,
a second indication to: (i) update the local set of parameters based on a common set of parameters, and (ii) use the updated local set of parameters in a subsequent run of the ML agent in the training mode; or
a third indication to: (i) terminate the training mode for the ML agent, and (ii) run the ML agent in an inference mode by using the common set of parameters,
wherein each of the second indication and the third indication comprises the common set of parameters, and the common set of parameters is obtained based on the local set of parameters of the network node and at least one other local set of parameters of at least one other network node.
14 . The network node of claim 13 , wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the network node to transmit the local set of parameters via an ML agent-specific signalling interface.
15 . The network node of claim 13 - or 14 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity to:
after each run of the ML agent in the training mode, receive a request for the local set of parameters from the ML orchestrator entity; and
in response to the request, transmit the local set of parameters to the ML orchestrator entity.
16 . The network node of claim 13 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the ML orchestrator entity to:
after each run of the ML agent in the training mode, transmit a request for the common set of parameters to the ML orchestrator entity; and
in response to the request, receive the second indication or the third indication from the ML orchestrator entity.
17 . The network node of claim 13 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the network node to receive, together with the second indication, a time instant from which the common set of parameters is to be used for updating the local set of parameters at the network node before the subsequent run of the ML agent is initiated.
18 . The network node of claim 13 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the network node to receive, together with the third indication, a time instant from which the common set of parameters is to be used in the inference mode of the ML agent at the network node.
19 . The network node of claim 13 , wherein the ML agent comprises a reinforcement learning (RL) agent configured to run in an exploration mode as the training mode and in an exploitation mode as the inference mode.
20 . The network node of claim 19 , wherein the RL agent is based on a Q-learning approach, and the at least one memory and the computer program code are configured to, with the at least one processor, cause the network node to present the local set of parameters as a local Q-table, and wherein the common set of parameters is presented as a common Q-table.
21 . A method for operating a machine-learning (ML) orchestrator entity in a wireless communication network, said method comprising:
grouping a set of network nodes in the wireless communication network into at least one node cluster based on at least one radio condition of a set of cells served by the set of network nodes, each network node from the set of network nodes having an ML agent installed thereon, the ML agent being configured to run based on a radio measurement in a training mode and in an inference mode; and
for each node cluster from the at least one node cluster:
transmitting, to each network node of the node cluster, a first indication to iteratively run the ML agent in the training mode, each run of the ML agent in the training mode yielding a local set of parameters for the ML agent; and
after each run of the ML agents in the training mode:
receiving at least two local sets of parameters from at least two network nodes of the node cluster;
based on the at least two local sets of parameters, obtaining a common set of parameters;
based on the common set of parameters, checking whether a termination criterion for the training mode is met; and
if the termination criterion for the training mode is not met, transmitting, to each network node of the node cluster, a second indication to: (i) update the local set of parameters based on the common set of parameters, and (ii) use the updated local set of parameters in a subsequent run of the ML agent in the training mode, the second indication comprising the common set of parameters.
22 . A method for operating a network node in a wireless communication network, said method comprising:
receiving, from a machine-learning (ML) orchestrator entity, a first indication to iteratively run an ML agent installed on the network node in a training mode, each run of the ML agent in the training mode yielding a local set of parameters for the ML agent; and
after each run of the ML agent in the training mode:
transmitting the local set of parameters to the ML orchestrator entity; and
receiving, from the ML orchestrator entity,
a second indication to: (i) update the local set of parameters based on a common set of parameters, and (ii) use the updated local set of parameters in a subsequent run of the ML agent in the training mode; or
a third indication to: (i) terminate the training mode for the ML agent, and (ii) run the ML agent in an inference mode by using the common set of parameters,
wherein each of the second indication and the third indication comprises the common set of parameters, and the common set of parameters is obtained based on the local set of parameters of the network node and at least one other local set of parameters of at least one other network node.
23 . A non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores computer code thereon which, when executed by at least one processor, causes the at least one processor to perform the method according to claim 21 .
24 . A non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores computer code thereon which, when executed by at least one processor, causes the at least one processor to perform the method according to claim 22 .