Transformation for covariate shift of grasp neural networks
A covariate shift generally refers to the change of the distribution of the input data (e.g., noise distribution) between the training and inference regimes. Such covariate shifts can degrade the performance grasping neural networks, and thus robotic grasping operations. As described herein, an output of a grasp neural network can be transformed, so as to determine appropriate locations on a given object for a robot or autonomous machine to grasp.
1 . A method of grasping an object, the method comprising:
capturing an image of the object within a physical environment, each of the object and the physical environment defining a plurality of locations, the image defining a plurality of pixels representative of the plurality of locations of the object and the physical environment;
sending the image of the object within the physical environment to a neural network;
based on the image, the neural network generating an output that defines respective grasp scores associated with the plurality of pixels, the grasp scores indicating a respective grasp quality at the locations of the object represented by the respective pixel;
transforming the output of the neural network, wherein transforming the output of the neural network comprises:
sorting the grasp scores in order of grasp quality and, for grasp locations associated with adjacent grasp scores in the sorted list, determining respective distances, comparing the distances to a minimum distance, and rejecting a highest-ranked grasp location as an outlier based on a determination that the distance to a next highest-ranked location exceeds the minimum distance; and
extracting a grasp point and surrounding points to generate a point cloud, fitting a plane through the points by least squares, computing residuals of the points to the plane, comparing the residuals to a predetermined threshold to detect whether an edge is near the grasp point, and, based on a determination that an edge is detected, moving the grasp point away from the edge by a predetermined distance based on a size or geometry of an end effector;
based on transforming the output of the neural network, selecting a grasp location of the plurality of locations of the object;
grasping the object, by an autonomous machine comprising an end effector, at the grasp location.
2 . The method as recited in claim 1 , wherein transforming the output of the neural network further comprises:
identifying a first grasp score of the grasp scores that indicates a high grasp quality as compared to the other grasp scores, the first grasp score associated with a first location of the plurality of locations; and
rejecting the first grasp score that indicates the high grasp quality, such that the grasp location that is selected is associated with one of the grasp scores other than the first grasp score.
3 . The method as recited in claim 2 , wherein rejecting the first grasp score comprises:
determining that the first location defines a point on the physical environment outside of the object.
4 . The method as recited in claim 2 , wherein rejecting the first grasp score further comprises:
sorting the grasp scores in an order from high grasp quality to low grasp quality, so as to define a sorted list of grasp scores that defines at least one grasp score proximate to the first grasp score; and
determining that the respective locations associated with the at least one grasp score proximate to the first grasp score are greater than a predetermined distance away from the first location associated with the first grasp score.
5 . The method as recited in claim 2 , wherein the object defines at least one edge, the first grasp score is associated with a first location, and rejecting the first grasp score further comprises:
determining a first distance from the first location to the at least one edge;
comparing the first distance to a predetermined threshold; and
determining that the first distance is less than the predetermined threshold.
6 . The method as recited in claim 5 , wherein the autonomous machine comprises an end effector configured to grasp the object, and the predetermined threshold is based on the end effector.
7 . The method as recited in claim 6 , wherein the predetermined threshold is based on a size of the end effector.
8 . The method as recited in claim 7 , wherein the end effector includes a suction cup having a diameter, and the predetermined threshold is based on the diameter.
9 . The method as recited in claim 2 , wherein transforming the output of the neural network further comprises:
identifying a first grasp score that indicates a high grasp quality as compared to the other grasp scores, the first grasp score associated with a first location of the plurality of locations;
determining a first distance between an edge of the object and the first location;
based on the first distance, determining a second distance; and
selecting the grasp location such that the grasp location is the second distance away from the first location.
10 . A non-transitory computer-readable storage medium including instructions that, when processed by a computing system, configure the computing system to perform the method according to claim 1 .
11 . The method of claim 1 , wherein the autonomous machine comprises an end effector that includes a suction cup, and wherein transforming the output comprises comparing distances between locations associated with grasp scores to a minimum distance that relates to size properties of the end effector and is defined based on a diameter of the suction cup, and selecting the grasp location by moving a grasp point away from an edge by a distance that is at least half the diameter of the suction cup.