Method and system for grasping an object
The method for increasing the accuracy of grasping an object can include: labelling an image based on an attempted object grasp by a robot and generating a trained graspability network using the labelled images. The method can additionally or alternatively include determining a grasp point using the trained graspability network; executing an object grasp at the grasp point S 400 ; and/or any other suitable elements.
1. A method comprising:
receiving an image of a plurality of objects;
generating a graspability heat map comprising a respective grasp score for each of a set of pixels of the image, wherein each of the respective grasp scores indicates a predicted likelihood of successfully grasping an object at a corresponding grasp point associated with the corresponding pixel;
selecting a grasp point based on the graspability heat map, wherein the grasp point is associated with a target object determined from the plurality of objects; and
determining control instructions for a robotic manipulator to grasp the target object based on the selected grasp point.
2. The method of claim 1 , wherein selecting a grasp point comprises:
selecting a set of grasp points from the graspability heat map based on the respective grasp score; and
selecting the grasp point from the set of grasp points.
3. The method of claim 2 , wherein the grasp point is selected from the set of grasp points using a grasp selector, wherein the grasp selector is configured to adjust the grasp score for the grasp point based on a set of heuristics, the set of heuristics comprising:
a collision avoidance rule; and
a failure hysteresis rule for a sliding temporal window of recent grasp attempts.
4. The method of claim 1 , further comprising: determining object parameters from the image using an object detector, wherein the control instructions are further determined based on an object parameter for the target object.
5. The method of claim 4 , wherein the neural network is trained using training images, grasp points selected based on object parameters extracted from the training images by the object detector, and grasp success or failure for each of the selected grasp points.
6. The method of claim 4 , further comprising: updating the neural network based on the image, the selected grasp point, and an outcome of a grasp attempt according to the control instructions.
7. The method of claim 1 , wherein the neural network comprises a convolutional neural network which is predetermined for a specific object type, wherein the plurality of objects consists of the specific object type.
8. The method of claim 1 , wherein the plurality of objects comprises a first object overlapping a second object.
9. A system comprising:
one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving an image of a plurality of objects;
generating a graspability heat map comprising a respective grasp score for each of a set of pixels of the image, wherein each of the respective grasp scores indicates a predicted likelihood of successfully grasping an object at a corresponding grasp point associated with the corresponding pixel;
selecting a grasp point based on the graspability heat map, wherein the grasp point is associated with a target object determined from the plurality of objects; and
determining control instructions for a robotic manipulator to grasp the target object based on the selected grasp point.
10. The system of claim 9 , wherein selecting a grasp point comprises:
selecting a set of grasp points from the graspability heat map based on the respective grasp score; and
selecting the grasp point from the set of grasp points.
11. The system of claim 9 , wherein the grasp point is selected from the set of grasp points using a grasp selector, wherein the grasp selector is configured to adjust the grasp score for the grasp point based on a set of heuristics, the set of heuristics comprising:
a collision avoidance rule; and
a failure hysteresis rule for a sliding temporal window of recent grasp attempts.
12. The system of claim 9 , wherein the operations further comprise: determining object parameters from the image using an object detector, wherein the control instructions are further determined based on an object parameter for the target object.
13. The system of claim 12 , wherein the neural network is trained using training images, grasp points selected based on object parameters extracted from the training images by the object detector, and grasp success or failure for each of the selected grasp points.
14. The system of claim 12 , wherein the operations further comprise: updating the neural network based on the image, the selected grasp point, and an outcome of a grasp attempt according to the control instructions.
15. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving an image of a plurality of objects;
generating a graspability heat map comprising a respective grasp score for each of a set of pixels of the image, wherein each of the respective grasp scores indicates a predicted likelihood of successfully grasping an object at a corresponding grasp point associated with the corresponding pixel;
selecting a grasp point based on the graspability heat map, wherein the grasp point is associated with a target object determined from the plurality of objects; and
determining control instructions for a robotic manipulator to grasp the target object based on the selected grasp point.
16. The one or more non-transitory computer storage media of claim 15 , wherein selecting a grasp point comprises:
selecting a set of grasp points from the graspability heat map based on the respective grasp score; and
selecting the grasp point from the set of grasp points.
17. The one or more non-transitory computer storage media of claim 15 , wherein the grasp point is selected from the set of grasp points using a grasp selector, wherein the grasp selector is configured to adjust the grasp score for the grasp point based on a set of heuristics, the set of heuristics comprising:
a collision avoidance rule; and
a failure hysteresis rule for a sliding temporal window of recent grasp attempts.
18. The one or more non-transitory computer storage media of claim 15 , wherein the operations further comprise: determining object parameters from the image using an object detector, wherein the control instructions are further determined based on an object parameter for the target object.
19. The one or more non-transitory computer storage media of claim 18 , wherein the neural network is trained using training images, grasp points selected based on object parameters extracted from the training images by the object detector, and grasp success or failure for each of the selected grasp points.
20. The one or more non-transitory computer storage media of claim 18 , wherein the operations further comprise: updating the neural network based on the image, the selected grasp point, and an outcome of a grasp attempt according to the control instructions.