Environment navigation using reinforcement learning
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. In one aspect, a method of training an action selection policy neural network for use in selecting actions to be performed by an agent navigating through an environment to accomplish one or more goals comprises: receiving an observation image characterizing a current state of the environment; processing, using the action selection policy neural network, an input comprising the observation image to generate an action selection output; processing, using a geometry-prediction neural network, an intermediate output generated by the action selection policy neural network to predict a value of a feature of a geometry of the environment when in the current state; and backpropagating a gradient of a geometry-based auxiliary loss into the action selection policy neural network to determine a geometry-based auxiliary update for current values of the network parameters.
1. A computer-implemented method of training an action selection neural network having a plurality of network parameters for use in selecting actions to be performed by an agent navigating through an environment to accomplish one or more goals, the method comprising:
receiving an observation characterizing a current state of the environment;
processing, using the action selection neural network and in accordance with current values of the network parameters, an input comprising the observation to generate an action selection output for use in controlling the agent to perform the actions;
processing, using a loop closure prediction neural network, an intermediate output generated by the action selection neural network to predict a value of a loop closure feature that characterizes whether, when the environment is in the current state, the agent has returned to a location in the environment that the agent has already visited;
determining an actual value of the loop closure feature; and
determining a loop closure-based auxiliary update for the current values of the network parameters of the action selection neural network based on an error between the prediction of the value of the loop closure feature and the actual value of the loop closure feature, comprising:
determining a gradient of a loop closure-based auxiliary loss that is based on the error between the prediction of the value of the loop closure feature and the actual value of the loop closure feature; and
backpropagating the gradient of the loop closure-based auxiliary loss into the action selection neural network to determine the loop closure-based auxiliary update for the current values of the network parameters.
2. The method of claim 1 , wherein determining the actual value of the loop closure feature comprises:
determining whether a current location of the agent in the environment at a current time step is within a first threshold distance of a particular previous location of the agent in the environment at a particular previous time step; and
determining whether a location of the agent in the environment deviated from the current location of the agent at a time step between the particular previous time step and the current time step by at least a second threshold distance.
3. The method of claim 1 , wherein the action selection neural network comprises:
a convolutional encoder neural network that generates an encoded representation of the input,
an intermediate neural network that processes the encoded representation of the input to generate an intermediate representation, and
an output neural network that processes the intermediate representation to generate the action selection output.
4. The method of claim 3 , wherein the intermediate output comprises the encoded representation.
5. The method of claim 3 , wherein the intermediate output comprises the intermediate representation.
6. The method of claim 3 , wherein the intermediate neural network comprises a recurrent neural network.
7. The method of claim 3 , wherein the intermediate neural network comprises a feedforward neural network.
8. The method of claim 1 , further comprising:
selecting an action to be performed by the agent when the environment is in the current state using the action selection output;
receiving a reward that is based on progress toward accomplishing the one or more goals; and
determining a main update for the current values of the network parameters of the action selection neural network based at least in part on the reward.
9. The method of claim 8 , further comprising:
using the loop closure-based auxiliary update and the main update to update the current values of the network parameters of the action selection neural network.
10. The method of claim 1 , further comprising:
updating the current values of the network parameters of the action selection neural network using the loop closure-based auxiliary update to generate updated parameter values; and
using the action selection neural network to select an action to be performed by the agent in accordance with the updated parameter values.
11. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training an action selection neural network having a plurality of network parameters for use in selecting actions to be performed by an agent navigating through an environment to accomplish one or more goals, the operations comprising:
receiving an observation characterizing a current state of the environment;
processing, using the action selection neural network and in accordance with current values of the network parameters, an input comprising the observation to generate an action selection output for use in controlling the agent to perform the actions;
processing, using a loop closure prediction neural network, an intermediate output generated by the action selection neural network to predict a value of a loop closure feature that characterizes whether, when the environment is in the current state, the agent has returned to a location in the environment that the agent has already visited;
determining an actual value of the loop closure feature; and
determining a loop closure-based auxiliary update for the current values of the network parameters of the action selection neural network based on an error between the prediction of the value of the loop closure feature and the actual value of the loop closure feature, comprising:
determining a gradient of a loop closure-based auxiliary loss that is based on the error between the prediction of the value of the loop closure feature and the actual value of the loop closure feature; and
backpropagating the gradient of the loop closure-based auxiliary loss into the action selection neural network to determine the loop closure-based auxiliary update for the current values of the network parameters.
12. The system of claim 11 , wherein determining the actual value of the loop closure feature comprises:
determining whether a current location of the agent in the environment at a current time step is within a first threshold distance of a particular previous location of the agent in the environment at a particular previous time step; and
determining whether a location of the agent in the environment deviated from the current location of the agent at a time step between the particular previous time step and the current time step by at least a second threshold distance.
13. The system of claim 11 , wherein the action selection neural network comprises:
a convolutional encoder neural network that generates an encoded representation of the input,
an intermediate neural network that processes the encoded representation of the input to generate an intermediate representation, and
an output neural network that processes the intermediate representation to generate the action selection output.
14. The system of claim 13 , wherein the intermediate output comprises the encoded representation.
15. The system of claim 13 , wherein the intermediate output comprises the intermediate representation.
16. The system of claim 13 , wherein the intermediate neural network comprises a recurrent neural network.
17. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an action selection neural network having a plurality of network parameters for use in selecting actions to be performed by an agent navigating through an environment to accomplish one or more goals, the operations comprising:
receiving an observation characterizing a current state of the environment;
processing, using the action selection neural network and in accordance with current values of the network parameters, an input comprising the observation to generate an action selection output for use in controlling the agent to perform the actions;
processing, using a loop closure prediction neural network, an intermediate output generated by the action selection neural network to predict a value of a loop closure feature that characterizes whether, when the environment is in the current state, the agent has returned to a location in the environment that the agent has already visited;
determining an actual value of the loop closure feature; and
determining a loop closure-based auxiliary update for the current values of the network parameters of the action selection neural network based on an error between the prediction of the value of the loop closure feature and the actual value of the loop closure feature, comprising:
determining a gradient of a loop closure-based auxiliary loss that is based on the error between the prediction of the value of the loop closure feature and the actual value of the loop closure feature; and
backpropagating the gradient of the loop closure-based auxiliary loss into the action selection neural network to determine the loop closure-based auxiliary update for the current values of the network parameters.
18. The non-transitory computer storage media of claim 17 , wherein determining the actual value of the loop closure feature comprises:
determining whether a current location of the agent in the environment at a current time step is within a first threshold distance of a particular previous location of the agent in the environment at a particular previous time step; and
determining whether a location of the agent in the environment deviated from the current location of the agent at a time step between the particular previous time step and the current time step by at least a second threshold distance.
19. The non-transitory computer storage media of claim 17 , wherein the action selection neural network comprises:
a convolutional encoder neural network that generates an encoded representation of the input,
an intermediate neural network that processes the encoded representation of the input to generate an intermediate representation, and
an output neural network that processes the intermediate representation to generate the action selection output.
20. The non-transitory computer storage media of claim 19 , wherein the intermediate output comprises the encoded representation.