Object pose neural network system
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium for predicting object pose. In one aspect, a method includes receiving an image of an object having one or more feature points; providing the image as an input to a neural network subsystem trained to receive images of objects and to generate an output including a heat map for each feature point; applying a differentiable transformation on each heat map to generate respective one or more feature coordinates for each feature point; providing the feature coordinates for each feature point as input to an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of an object; and receiving, at the output of the object pose solver, a predicted object pose for the object in the image.
1. A computer-implemented method comprising:
receiving an image;
providing the image as an input to a neural network system trained to generate an output comprising a plurality of heat maps corresponding to a plurality of feature points, each heat map representing a likelihood, for each image region of a plurality of image regions in the image, that the image region corresponds to a respective one of the plurality of feature points;
applying a differentiable transformation on each of the plurality of heat maps to generate a plurality of feature coordinates for the plurality of feature points, wherein applying the differentiable transformation on each heat map comprises:
applying a soft argmax function to one or more values of each respective heat map to generate a location of a measure of central tendency of values in the heat map; and
generating a single feature point location within the image region corresponding to the heat map based on the location of the measure of central tendency of values in the heat map; and
providing the plurality of feature coordinates for the plurality of feature points.
2. The method of claim 1 , wherein the image is of a robot, and wherein the method further comprises controlling the robot based on the plurality of feature coordinates for the plurality of feature points.
3. The method of claim 2 , wherein the plurality of feature points corresponds to a plurality of points of the robot.
4. The method of claim 1 , further comprising generating a measure of variance for each feature location from the plurality of feature coordinates of the measure of central tendency of values in the respective heat maps.
5. The method of claim 1 , further comprising:
applying a respective curve fitting procedure to each heat map, each curve fitting procedure for a heat map generating a function based on values in the heat map; and
computing the plurality of feature coordinates based on respective locations of peaks of the function output by the curve fitting procedure.
6. The method of claim 1 , wherein the image is of an object, wherein the method further comprises:
inputting the plurality of feature coordinates for the plurality of feature points into an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of the object; and
outputting the predicted object pose for the object.
7. The method of claim 6 , wherein the object pose solver uses a least squares regression analysis procedure.
8. The method of claim 6 , further comprising obtaining a ground truth pose for the object by:
obtaining respective ground truth object poses of the object based on pose information contained in each of one or more other images;
computing one or more respective relative displacements of the object using kinematic data associated with the image and respective kinematic data associated with each of the one or more other images; and
computing an interpolated ground truth object pose of the object based on a respective ground truth pose of the object in each of the one or more other images and the one or more respective relative displacements.
9. The method of claim 1 , wherein the plurality of feature points each correspond to a visually distinguishable portion of a robotic arm.
10. A computer-implemented method comprising:
receiving an image, wherein the image is of an object;
providing the image as an input to a neural network system trained to generate an output comprising a plurality of heat maps corresponding to a plurality of feature points, each heat map representing a likelihood, for each image region of a plurality of image regions in the image, that the image region corresponds to a respective one of the plurality of feature points;
applying a differentiable transformation on each of the plurality of heat maps to generate a plurality of feature coordinates for the plurality of feature points;
providing the plurality of feature coordinates for the plurality of feature points;
obtaining a ground truth pose for the object by:
obtaining respective ground truth object poses of the object based on pose information contained in each of one or more other images;
computing one or more respective relative displacements of the object using kinematic data associated with the image and respective kinematic data associated with each of the one or more other images; and
computing an interpolated ground truth object pose of the object based on a respective ground truth pose of the object in each of the one or more other images and the one or more respective relative displacements;
inputting the plurality of feature coordinates for the plurality of feature points into an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of the object; and
outputting the predicted object pose for the object.