Object pose neural network system

1. A computer-implemented method comprising:

obtaining a pair of images of a particular environment of an object, each of the pair of images being associated with respective kinematic data;

computing, for each of the pair of images, a respective predicted object pose in the respective image, wherein computing the respective predicted object pose in the respective image comprises:

providing the respective image as an input to a neural network system configured to receive images of objects and to generate an output comprising a heat map for each of the one or more feature points, wherein each heat map represents a likelihood for each image region of a plurality of image regions in the image that the image region corresponds to a respective one of the one or more feature points;

applying a differentiable transformation on each heat map to generate one or more respective feature coordinates for each of the one or more feature points;

providing the one or more feature coordinates for each of the one or more feature points as input to an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of an object; and

receiving, at the output of the object pose solver, a predicted object pose for the object in the respective image;

computing a difference between a displacement of the predicted object poses and a displacement indicated by the kinematic data; and

training a neural network system using an error between the displacement of the predicted object poses and a displacement indicated by the kinematic data.

2. The method of claim 1 , wherein each respective kinematic data associated with a respective image is determined based on respective values of one or more connecting angles of the object at a time of the respective image.

3. The method of claim 1 , wherein the one or more feature points each correspond to a feature marker in an environment of the object.

4. The method of claim 1 , wherein the neural network subsystem includes one or more convolutional neural network layers.

5. The method of claim 1 , wherein applying the differentiable transformation on each heat map comprises:

applying a soft argmax function to one or more values of each respective heat map to generate a location of a measure of central tendency of values in the heat map; and

generating the respective one or more feature coordinates for the respective feature point corresponding to each respective heat map based on the location of a measure of central tendency of values in the heat map.

6. The method of claim 5 , further comprising:

generating a measure of variance for each feature coordinate from the computed locations of the measure of central tendency of values in the respective heat maps; and

providing the measure of variance for each feature coordinate to the object pose solver, wherein the object pose solver computes the predicted object pose by giving more weight to feature coordinates having a lower variance.

7. The method of claim 1 , wherein applying the differentiable transformation to generate one or more respective feature coordinates for each of the one or more feature points comprises applying a curve fitting procedure to the values in the heat maps and computing the respective feature coordinates based on respective locations of peaks of the curves output by the curve fitting procedure.

8. The method of claim 1 , wherein the object pose solver is configured to compute a predicted object pose for the object from the feature coordinates using a least squares regression analysis procedure.

9. The method of claim 1 , wherein the object is an end effector of a robotic arm.

10. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

obtaining a pair of images of a particular environment of an object, each of the pair of images being associated with respective kinematic data;

computing, for each of the pair of images, a respective predicted object pose in the respective image, wherein computing the respective predicted object pose in the respective image comprises:

providing the respective image as an input to a neural network system configured to receive images of objects and to generate an output comprising a heat map for each of the one or more feature points, wherein each heat map represents a likelihood for each image region of a plurality of image regions in the image that the image region corresponds to a respective one of the one or more feature points;

applying a differentiable transformation on each heat map to generate one or more respective feature coordinates for each of the one or more feature points;

providing the one or more feature coordinates for each of the one or more feature points as input to an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of an object; and

receiving, at the output of the object pose solver, a predicted object pose for the object in the respective image;

computing a difference between a displacement of the predicted object poses and a displacement indicated by the kinematic data; and

training a neural network system using an error between the displacement of the predicted object poses and a displacement indicated by the kinematic data.

11. The system of claim 10 , wherein each respective kinematic data associated with a respective image is determined based on respective values of one or more connecting angles of the object at a time of the respective image.

12. The system of claim 10 , wherein the one or more feature points each correspond to a feature marker in an environment of the object.

13. The system of claim 10 , wherein the neural network subsystem includes one or more convolutional neural network layers.

14. The system of claim 10 , wherein applying the differentiable transformation on each heat map comprises:

applying a soft argmax function to one or more values of each respective heat map to generate a location of a measure of central tendency of values in the heat map; and

generating the respective one or more feature coordinates for the respective feature point corresponding to each respective heat map based on the location of a measure of central tendency of values in the heat map.

15. The system of claim 14 , wherein the operations further comprise:

generating a measure of variance for each feature coordinate from the computed locations of the measure of central tendency of values in the respective heat maps; and

providing the measure of variance for each feature coordinate to the object pose solver, wherein the object pose solver computes the predicted object pose by giving more weight to feature coordinates having a lower variance.

16. The system of claim 10 , wherein applying the differentiable transformation to generate one or more respective feature coordinates for each of the one or more feature points comprises applying a curve fitting procedure to the values in the heat maps and computing the respective feature coordinates based on respective locations of peaks of the curves output by the curve fitting procedure.

17. The system of claim 10 , wherein the object pose solver is configured to compute a predicted object pose for the object from the feature coordinates using a least squares regression analysis procedure.

18. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining a pair of images of a particular environment of an object, each of the pair of images being associated with respective kinematic data;

computing, for each of the pair of images, a respective predicted object pose in the respective image, wherein computing the respective predicted object pose in the respective image comprises:

providing the respective image as an input to a neural network system configured to receive images of objects and to generate an output comprising a heat map for each of the one or more feature points, wherein each heat map represents a likelihood for each image region of a plurality of image regions in the image that the image region corresponds to a respective one of the one or more feature points;

applying a differentiable transformation on each heat map to generate one or more respective feature coordinates for each of the one or more feature points;

providing the one or more feature coordinates for each of the one or more feature points as input to an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of an object; and

receiving, at the output of the object pose solver, a predicted object pose for the object in the respective image;

computing a difference between a displacement of the predicted object poses and a displacement indicated by the kinematic data; and

training a neural network system using an error between the displacement of the predicted object poses and a displacement indicated by the kinematic data.