Noisy neural network layers with noise parameters
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting an action to be performed by a reinforcement learning agent. The method includes obtaining an observation characterizing a current state of an environment. For each layer parameter of each noisy layer of a neural network, a respective noise value is determined. For each layer parameter of each noisy layer, a noisy current value for the layer parameter is determined from a current value of the layer parameter, a current value of a corresponding noise parameter, and the noise value. A network input including the observation is processed using the neural network in accordance with the noisy current values to generate a network output for the network input. An action is selected from a set of possible actions to be performed by the agent in response to the observation using the network output.
1. A method of training a neural network,
wherein the neural network is configured to receive a network input and to process the network input to generate a network output,
wherein the neural network comprises a plurality of layers each having respective layer parameters,
wherein one or more of the layers are noisy layers, and
wherein the method comprises:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining a training network input;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value, comprising:
generating a noise modifier based on the current value of the corresponding noise parameter and the noise value; and
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter;
processing the training input using the neural network in accordance with the noisy current values to generate a network output for the training input;
determining a gradient of an objective function that depends on the network output with respect to the current values of the layer parameters and the current values of the noise parameters; and
determining an update to the current values of the layer parameters and the current values of the noise parameters from the gradient.
2. The method of claim 1 , wherein
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter comprises:
adding the noise modifier and the current value of the layer parameter to generate the noisy current value for the layer parameter.
3. The method of claim 1 , wherein determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution comprises:
for each noisy layer, sampling a predetermined number of random values from one or more predetermined probability distributions; and
for each layer parameter of each noisy layer, generating the noise value for the layer parameter by combining two or more of the sampled random values for the noisy layer.
4. The method of claim 1 , further comprising selecting an action to be performed by a reinforcement learning agent interacting with an environment based on the network output.
5. The method of claim 4 , wherein the objective function comprises a reinforcement learning objective function.
6. The method of claim 4 , wherein the network input comprises an observation characterizing a state of the environment and an action from a set of actions, and wherein the network output is an estimate of a return received if the reinforcement learning agent performs the action in response to the observation.
7. The method of claim 4 , wherein the network input comprises an observation characterizing a state of the environment and the network output defines a likelihood distribution over actions in a set of possible actions to be performed by the agent in response to the observation.
8. The method of claim 1 , wherein the plurality of layers also includes one or more layers that are not noisy layers, and wherein processing the training input using the neural network comprises processing the training input in accordance with the noisy current values and the current values of the layer parameters of the layers that are not noise layers to generate the network output for the training input.
9. The method of claim 1 , wherein the one or more noisy layers include one or more fully-connected layers.
10. The method of claim 1 , wherein the one or more noisy layers include one or more convolutional layers.
11. The method of claim 1 , wherein the one or more noisy layers include one or more recurrent neural network layers.
12. A method of selecting an action to be performed by a reinforcement learning agent interacting with an environment using a neural network, the neural network having a plurality of layers, each layer having a plurality of layer parameters, and the plurality of layers including one or more noisy layers, the method comprising:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining an observation characterizing a current state of the environment;
generating a network input for the neural network comprising the observation;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value, comprising:
generating a noise modifier based on the current value of the corresponding noise parameter and the noise value; and
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter;
processing the network input using the neural network in accordance with the noisy current values to generate a network output for the network input; and
selecting an action from a set of possible actions to be performed by the agent in response to the observation using the network output;
wherein for each noisy layer, the layer parameters and the corresponding noise parameters have been trained to maximize a cumulative measure of rewards received by selecting actions to be performed by the reinforcement learning agent using network outputs generated by the neural network.
13. The method of claim 12 , wherein the network input further comprises a given action from the set of possible actions, and wherein the network output is an estimate of a return received if the reinforcement learning agent performs the given action in response to the observation.
14. The method of claim 12 , wherein the network output defines a likelihood distribution over the actions in the set of possible actions.
15. The method of claim 12 , wherein the network output identifies the action from the set of possible actions.
16. The method of claim 1 , wherein the objective function additionally depends on a target output for the training network input.
17. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network,
wherein the neural network is configured to receive a network input and to process the network input to generate a network output,
wherein the neural network comprises a plurality of layers each having respective layer parameters,
wherein one or more of the layers are noisy layers, and
wherein the operations comprise:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining a training network input;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value, comprising:
generating a noise modifier based on the current value of the corresponding noise parameter and the noise value; and
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter;
processing the training input using the neural network in accordance with the noisy current values to generate a network output for the training input;
determining a gradient of an objective function that depends on the network output with respect to the current values of the layer parameters and the current values of the noise parameters; and
determining an update to the current values of the layer parameters and the current values of the noise parameters from the gradient.
18. The system of claim 17 , wherein
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter comprises:
adding the noise modifier and the current value of the layer parameter to generate the noisy current value for the layer parameter.
19. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network,
wherein the neural network is configured to receive a network input and to process the network input to generate a network output,
wherein the neural network comprises a plurality of layers each having respective layer parameters,
wherein one or more of the layers are noisy layers, and
wherein the operations comprise:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, iii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining a training network input;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value, comprising:
generating a noise modifier based on the current value of the corresponding noise parameter and the noise value; and
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter;
processing the training input using the neural network in accordance with the noisy current values to generate a network output for the training input;
determining a gradient of an objective function that depends on the network output with respect to the current values of the layer parameters and the current values of the noise parameters; and
determining an update to the current values of the layer parameters and the current values of the noise parameters from the gradient.
20. The non-transitory computer storage media of claim 19 , wherein
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter comprises:
adding the noise modifier and the current value of the layer parameter to generate the noisy current value for the layer parameter.
21. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for selecting an action to be performed by a reinforcement learning agent interacting with an environment using a neural network, the neural network having a plurality of layers, each layer having a plurality of layer parameters, and the plurality of layers including one or more noisy layers, the operations comprising:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining an observation characterizing a current state of the environment;
generating a network input for the neural network comprising the observation;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value, comprising:
generating a noise modifier based on the current value of the corresponding noise parameter and the noise value; and
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter;
processing the network input using the neural network in accordance with the noisy current values to generate a network output for the network input; and
selecting an action from a set of possible actions to be performed by the agent in response to the observation using the network output;
wherein for each noisy layer, the layer parameters and the corresponding noise parameters have been trained to maximize a cumulative measure of rewards received by selecting actions to be performed by the reinforcement learning agent using network outputs generated by the neural network.
22. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting an action to be performed by a reinforcement learning agent interacting with an environment using a neural network, the neural network having a plurality of layers, each layer having a plurality of layer parameters, and the plurality of layers including one or more noisy layers, the operations comprising:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining an observation characterizing a current state of the environment;
generating a network input for the neural network comprising the observation;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value, comprising:
generating a noise modifier based on the current value of the corresponding noise parameter and the noise value; and
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter;
processing the network input using the neural network in accordance with the noisy current values to generate a network output for the network input; and
selecting an action from a set of possible actions to be performed by the agent in response to the observation using the network output;
wherein for each noisy layer, the layer parameters and the corresponding noise parameters have been trained to maximize a cumulative measure of rewards received by selecting actions to be performed by the reinforcement learning agent using network outputs generated by the neural network.