Methods and systems for training a neural network using sequential Monte Carlo processing
A computer-implemented method, a machine learning system, and non-transitory computer-readable storage medium for training a neural network are provided. The neural network is used to instruct an agent to select actions for interacting with an environment to determine a solution to a specified problem. In the computer-implemented method a state signal representing a current state of the environment is received. A Sequential Monte Carlo process is then used to perform a search to determine target action selection data associated with the current state of the environment. This target action selection data is stored in association with the state signal and the current state of the environment is updated by providing an action selection signal based on the target action selection data. The Sequential Monte Carlo process involves generating a plurality of simulations using the neural network to determine the target action selection data.
1 . A computer-implemented method of training a neural network having a plurality of network parameter values, the neural network being used to instruct an agent to select actions for interacting with an environment to determine a solution to a specified problem,
wherein the neural network is adapted to receive a state signal representing a state of the environment and to process the state signal according to the plurality of network parameter values to generate action selection data representing an action selection policy for selecting an action to be performed by the agent in response to the state signal,
wherein the computer-implemented method comprises:
receiving a state signal representing a current state of the environment;
using a Sequential Monte Carlo process to perform a search to determine target action selection data associated with the current state of the environment, the Sequential Monte Carlo process comprising:
generating a plurality of simulations each comprising a respective sequence of states and selected actions, each respective sequence starting at a first state of the environment and ending in a respective second state of the environment wherein the actions are selected using the neural network, and wherein at each state a weight value associated with the respective simulation is updated, wherein generating the plurality of simulations comprises:
monitoring weight values associated with the plurality of simulations;
selecting a first subset of the plurality of simulations for regeneration based on the monitoring; and
regenerating the first subset of the plurality of simulations by cloning a second subset of the plurality of simulations which were not selected for regeneration;
storing data indicative of the plurality of simulations in a buffer as the plurality of simulations are generated; and
determining the target action selection data by evaluating the weight values associated with the plurality of simulations;
storing the determined target action selection data in association with the state signal representing the current state of the environment for updating the network parameter values; and
updating the current state of the environment by providing an action selection signal based on the determined target action selection data.
2 . The computer-implemented method according to claim 1 , wherein the computer-implemented method is repeated until the environment is in a terminal state, and wherein the computer-implemented method further comprises training the neural network using the stored target action selection data based on an evaluation of the terminal state of the environment.
3 . The computer-implemented method according to claim 1 , wherein the buffer has a fixed capacity and storing data indicative of the plurality of simulations in the buffer includes replacing data previously stored in the buffer with data indicative of the plurality of simulations.
4 . The computer-implemented method according to claim 1 , wherein the neural network is adapted to generate predicted expected reward data in response to the received state signal by processing the received state signal according to the network parameter values, and wherein the action selection data comprises predicted expected reward data generated by the neural network.
5 . The computer-implemented method according to claim 1 , wherein selecting actions using the neural network to generate a said simulation of the plurality of simulations comprises at a said state of the said simulation using the neural network to determine at least a distribution over available actions and selecting an available action based on at least the distribution.
6 . The computer-implemented method according to claim 1 , wherein selecting the first subset of the plurality of simulations is triggered based on the monitoring.
7 . The computer-implemented method according to claim 1 , wherein the monitoring comprises comparing the weight values with a predetermined threshold.
8 . The computer-implemented method according to claim 1 , wherein the monitoring comprises updating a plurality of variance measures associated with the weight values and comparing the plurality of variance measures to a predetermined threshold.
9 . The computer-implemented method according to claim 1 , wherein the weight value for a said simulation of the plurality of simulations is updated based on predicted expected reward data for each state of the said simulation.
10 . The computer-implemented method according to claim 1 , wherein each simulation is associated with an entropy value, the entropy value being iteratively updated according to the sequence of actions, and wherein the weight values of the simulations are updated according to their respective associated entropy value.
11 . A machine learning system for training a neural network, the neural network being used to instruct an agent to select actions for interacting with an environment to determine a solution to a specified problem, the system comprising:
a neural network having a plurality of network parameter values, the neural network being adapted to receive a state signal representing a state of an environment and to process the state signal according to the plurality of network parameter values to generate action selection data representing an action selection policy for selecting an action to be performed by the agent in response to the state signal;
a buffer;
a training engine adapted to perform an iterative process, the iterative process comprising, iteratively:
receiving a state signal representing a current state of the environment;
using a Sequential Monte Carlo process to perform a search to determine target action selection data associated with a current state of the environment, the Sequential Monte Carlo process comprising:
generating a plurality of simulations each comprising a respective sequence of states and selected actions, each respective sequence starting from a first state of the environment and ending in a respective second state of the environment wherein the actions are selected using the neural network, and wherein at each state a weight value associated with the respective simulation is updated, wherein generating the plurality of simulations comprises:
monitoring weight values associates with the plurality of simulations;
selecting a first subset of the plurality of simulations for regeneration based on the monitoring; and
regenerating the first subset of the plurality of simulations by cloning a second subset of the plurality of simulations which were not selected for regeneration;
storing data indicative of the plurality of simulations in the buffer as the plurality of simulations are generated; and
determining the target action selection data by evaluating the weight values associated with the plurality of simulations;
store the target action selection data in association with the state signal representing the first state of the environment for updating the network parameter values of the neural network; and
update the current state of the environment by providing an action selection signal according to the target action selection data.
12 . A non-transitory computer-readable storage medium comprising computer readable instructions stored thereon which, when executed by at least one processor, cause the at least one processor to perform the method according to claim 1 .