Optimizing policy controllers for robotic agents using image embeddings
There are provided systems, methods, and apparatus, for optimizing a policy controller to control a robotic agent that interacts with an environment to perform a robotic task. One of the methods includes optimizing the policy controller using a neural network that generates numeric embeddings of images of the environment and a demonstration sequence of demonstration images of another agent performing a version of the robotic task.
1. A method of optimizing a policy controller used to select actions to be performed by a robotic agent interacting with an environment to perform a specified task, the method comprising:
obtaining a demonstration sequence of demonstration images of another agent performing a version of the specified task;
for each respective demonstration image in the demonstration sequence, generating a respective demonstration embedding of the respective demonstration image by processing the respective demonstration image using a time contrastive neural network that has been trained on time-sequenced images of a training environment to minimize a loss that includes a difference between embeddings generated for co-occurring input images captured from different viewpoints or by different modalities;
obtaining a robot sequence of robot images of the robotic agent performing the specified task by performing actions selected using a current policy controller, wherein each robot image in the robot sequence corresponds to a respective demonstration image in the demonstration sequence;
for each respective robot image in the robot sequence, generating a respective robot embedding for the respective robot image by processing the respective robot image using the same time contrastive neural network that has been trained; and
updating the current policy controller by performing an iteration of a reinforcement learning technique to optimize a reward function that depends on, for each demonstration image, a distance between (i) the demonstration embedding that has been generated by processing the demonstration image using the time contrastive neural network and (ii) the robot embedding that has been generated by processing the corresponding robot image using the same time contrastive neural network.
2. The method of claim 1 , wherein the other agent is a human demonstrator.
3. The method of claim 1 , wherein the other agent is a different robotic agent.
4. The method of claim 3 , wherein the policy controller is a time-varying Gaussian controller.
5. The method of claim 1 , wherein the policy controller is a trajectory-centric controller.
6. The method of claim 1 , wherein the reinforcement learning technique is a technique that combines model-based and model-free algorithms.
7. The method of claim 1 , wherein the reward function includes a Euclidean distance term that is a square of the Euclidean distance between the demonstration embedding of the demonstration image and the robot embedding of the corresponding robot image.
8. The method of claim 1 , wherein the reward function includes a Huber-style loss term that is a square root of a sum between a constant value and a square of the Euclidean distance between the demonstration embedding of the demonstration image and the robot embedding of the corresponding robot image.
9. The method of claim 1 , wherein the reward function satisfies:
R ( v t ,w t )=−α∥ w t −v t ∥ 2 2 −β√{square root over (γ+∥ w t −v t ∥ 2 2 )},
wherein vt is the demonstration embedding of the demonstration image in a t-th position in the demonstration sequence, wt is the robot embedding of the robot image in a t-th position in the robot sequence, α and β are fixed weighting parameters, and γ is a small positive constant value.
10. The method of claim 1 , wherein the images in the demonstration sequence are captured from a different viewpoint than the images in the robot sequence.
11. The method of claim 10 , wherein images in the demonstration sequence are captured from a third-party view relative to the other agent and the images in the robot sequence are captured from a first-party view relative to the robotic agent.
12. The method of claim 1 , wherein the corresponding robot image is the robot image that is in the same position in the robot sequence as the demonstration image is in the demonstration sequence.
13. The method of claim 1 , wherein the time contrastive neural network has a plurality of network parameters and wherein the method further comprises:
training the time contrastive neural network to generate numeric embeddings, the training comprising:
obtaining a first image of the environment captured by a first modality;
obtaining a second image that is co-occurring with the first image and that is captured by a second, different modality;
obtaining a third image captured by the first modality that is not co-occurring with the first image;
determining a gradient of a triplet loss that uses the first image as an anchor example, the second image as a positive example, and the third image as a negative example; and
updating current values of the network parameters using the gradient of the triplet loss.
14. The method of claim 13 , wherein the first modality is a camera at a first viewpoint, and wherein the second modality is a camera at a second, different viewpoint.
15. The method of claim 13 , wherein the third image is within a predetermined time window of the first image.
16. The method of claim 15 , wherein obtaining the third image comprises:
selecting an image that is a hard negative relative to the first image from the images captured by the first modality that are within the predetermined time window of the first image.
17. The method of claim 13 , wherein determining the gradient of the triplet loss comprises:
processing the first image using the time contrastive neural network in accordance with the current values of the network parameters to generate a first embedding;
processing the second image using the time contrastive neural network in accordance with the current values of the network parameters to generate a second embedding;
processing the third image using the time contrastive neural network in accordance with the current values of the network parameters to generate a third embedding; and
determining the triplet loss from (i) a first distance between the first embedding and the second embedding and (ii) a second distance between the first embedding and the third embedding.
18. The method of claim 1 , further comprising controlling the robotic agent using the optimized policy controller.
19. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for optimizing a policy controller used to select actions to be performed by a robotic agent interacting with an environment to perform a specified task, the operations comprising:
obtaining a demonstration sequence of demonstration images of another agent performing a version of the specified task;
for each respective demonstration image in the demonstration sequence, generating a respective demonstration embedding of the respective demonstration image by processing the respective demonstration image using a time contrastive neural network that has been trained on time-sequenced images of a training environment to minimize a loss that includes a difference between embeddings generated for co-occurring input images captured from different viewpoints or by different modalities;
obtaining a robot sequence of robot images of the robotic agent performing the specified task by performing actions selected using a current policy controller, wherein each robot image in the robot sequence corresponds to a respective demonstration image in the demonstration sequence;
for each respective robot image in the robot sequence, generating a respective robot embedding for the respective robot image by processing the respective robot image using the same time contrastive neural network; and
updating the current policy controller by performing an iteration of a reinforcement learning technique to optimize a reward function that depends on, for each demonstration image, a distance between (i) the demonstration embedding that has been generated by processing the demonstration image using the time contrastive neural network and (ii) the robot embedding that has been generated by processing the corresponding robot image using the same time contrastive neural network.
20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for optimizing a policy controller used to select actions to be performed by a robotic agent interacting with an environment to perform a specified task, the operations comprising:
obtaining a demonstration sequence of demonstration images of another agent performing a version of the specified task;
for each respective demonstration image in the demonstration sequence, generating a respective demonstration embedding of the respective demonstration image by processing the respective demonstration image using a time contrastive neural network that has been trained on time-sequenced images of a training environment to minimize a loss that includes a difference between embeddings generated for co-occurring input images captured from different viewpoints or by different modalities;
obtaining a robot sequence of robot images of the robotic agent performing the specified task by performing actions selected using a current policy controller, wherein each robot image in the robot sequence corresponds to a respective demonstration image in the demonstration sequence;
for each respective robot image in the robot sequence, generating a respective robot embedding for the respective robot image by processing the respective robot image using the same time contrastive neural network; and
updating the current policy controller by performing an iteration of a reinforcement learning technique to optimize a reward function that depends on, for each demonstration image, a distance between (i) the demonstration embedding that has been generated by processing the demonstration image using the time contrastive neural network and (ii) the robot embedding that has been generated by processing the corresponding robot image using the same time contrastive neural network.