Control apparatus, control method and program

1. A control device that dynamically allocates a virtual network for providing a network service on a physical network by multi-agent deep reinforcement learning, the control device comprising:

one or more processors configured to:

select an agent g k that executes an action that represents allocation of the virtual network onto the physical network at a time point t from a plurality of agents;

observe a state s t at the time point t with an input of network configuration information of the physical network, network observation information indicating a physical resource amount of the physical network, and user demand information indicating an amount demanded for a physical resource generated by a communication action of a user terminal of the network service;

select and execute, on basis of an action value function Q that represents an expected value of a sum total of a reward received in future, an action a t k from possible actions for the agent g k ;

calculate a reward r t at the time point t with use of the action a t k , the state s t , and a state s t+1 at a time point t+1; and

update the action value function Q with use of the action a j k , the state s j , and the state s j+1 , where 1≤j≤t is satisfied.

2. The control device according to claim 1 , wherein:

a total number of virtual networks is N; and

the one or more processors are configured to select the agent g k that executes the action that represents the allocation of the virtual network that belongs to a k-th group out of groups each having a number of N/M virtual networks obtained by dividing the number of N virtual networks.

3. The control device according to claim 2 , wherein the one or more processors are configured to select the agent g k , where k=t mod M is satisfied.

4. The control device according to claim 1 , wherein the one or more processors are configured to:

store a tuple of the action a j k , the state s j , and the state s j+1 in a replay memory; and

randomly select the j, and updates the action value function Q with use of the action a j k , the state s j , and the state s j+1 included in a tuple corresponding to the j out of the tuple stored in the replay memory.

5. The control device according to claim 1 , wherein the reward r t becomes a higher value as a network usage efficiency defined by a usage rate of a server included in the physical network and a usage rate of a link included in the physical network becomes higher, and becomes a higher value as a number of changes of the physical resource allocated to the virtual network increases.

6. A control method for dynamically allocating a virtual network for providing a network service on a physical network by multi-agent deep reinforcement learning, the control method comprising:

selecting an agent g k that executes an action that represents allocation of the virtual network onto the physical network at a time point t from a plurality of agents;

observing a state s t at the time point t with an input of network configuration information of the physical network, network observation information indicating a physical resource amount of the physical network, and user demand information indicating an amount demanded for a physical resource generated by a communication action of a user terminal of the network service;

selecting and executing, on basis of an action value function Q that represents an expected value of a sum total of a reward received in future, an action a t k from possible actions for the agent g k ;

calculating a reward r t at the time point t with use of the action a t k , the state s t , and a state s t+1 at a time point t+1; and

learning procedure of updating the action value function Q with use of the action a j k , the state s j , and the state s j+1 , where 1≤j≤t is satisfied.

7. A non-transitory computer readable medium storing one or more instructions for causing a computer to function as a control device that dynamically allocates a virtual network for providing a network service on a physical network by multi-agent deep reinforcement learning to execute:

selecting an agent g k that executes an action that represents allocation of the virtual network onto the physical network at a time point t from a plurality of agents;

observing a state s t at the time point t with an input of network configuration information of the physical network, network observation information indicating a physical resource amount of the physical network, and user demand information indicating an amount demanded for a physical resource generated by a communication action of a user terminal of the network service;

selecting and executing, on basis of an action value function Q that represents an expected value of a sum total of a reward received in future, an action a t k from possible actions for the agent g k ;

calculating a reward r t at the time point t with use of the action a t k , the state s t , and a state s t+1 at a time point t+1; and

learning procedure of updating the action value function Q with use of the action a j k , the state s j , and the state s j+1 , where 1≤j≤t is satisfied.

8. The control method according to claim 6 , wherein:

a total number of virtual networks is N; and

the control method comprises: selecting the agent g k that executes the action that represents the allocation of the virtual network that belongs to a k-th group out of groups each having a number of N/M virtual networks obtained by dividing the number of N virtual networks.

9. The control method according to claim 8 , comprising:

selecting the agent g k , where k=t mod M is satisfied.

10. The control method according to claim 6 , comprising:

storing a tuple of the action a j k , the state s j , and the state s j +1 in a replay memory; and

randomly selecting the j, and updates the action value function Q with use of the action a j k , the state si, and the state s j+1 included in a tuple corresponding to the j out of the tuple stored in the replay memory.

11. The control method according to claim 6 , wherein the reward r t becomes a higher value as a network usage efficiency defined by a usage rate of a server included in the physical network and a usage rate of a link included in the physical network becomes higher, and becomes a higher value as a number of changes of the physical resource allocated to the virtual network increases.

12. The non-transitory computer readable medium according to claim 7 , wherein:

a total number of virtual networks is N; and

the one or more instructions cause the computer to execute: selecting the agent g k that executes the action that represents the allocation of the virtual network that belongs to a k-th group out of groups each having a number of N/M virtual networks obtained by dividing the number of N virtual networks.

13. The non-transitory computer readable medium according to claim 12 , wherein the one or more instructions cause the computer to execute:

selecting the agent g k , where k=t mod M is satisfied.

14. The non-transitory computer readable medium according to claim 7 , wherein the one or more instructions cause the computer to execute:

storing a tuple of the action a j k , the state s j , and the state s j+1 in a replay memory; and

randomly selecting the j, and updates the action value function Q with use of the action a j k , the state s j , and the state s j+1 included in a tuple corresponding to the j out of the tuple stored in the replay memory.

15. The non-transitory computer readable medium according to claim 7 , wherein the reward r t becomes a higher value as a network usage efficiency defined by a usage rate of a server included in the physical network and a usage rate of a link included in the physical network becomes higher, and becomes a higher value as a number of changes of the physical resource allocated to the virtual network increases.