Method and system for object grasping
A method for object grasping, including: determining features of a scene; determining candidate grasp locations; determining a set of candidate grasp proposals for the candidate grasp locations; optionally modifying a candidate grasp proposal of the set; determining grasp scores associated with the candidate grasp proposals; selecting a set of final grasp proposals based on the grasp scores; and executing a grasp proposal from the set of final grasp proposals.
1. A method, comprising:
determining a set of data for an object in a scene, each of the set of data comprising a respective keypoint and a corresponding object component identifier;
determining a set of candidate grasp locations for the object, using the set of data, wherein each candidate grasp location of the set of candidate grasp locations is associated with a respective occlusion score determined by a machine learning model trained on images labeled with predetermined occlusion scores for locations in the images;
determining one or more candidate grasp proposals based on the set of candidate grasp locations for grasping the object;
selecting a candidate grasp proposal from the one or more candidate grasp proposals as a final grasp proposal based on one or more criteria; and
storing the final grasp proposal in memory to be retrieved for controlling a robot to grasp the object.
2. The method of claim 1 , wherein one of the one or more candidate grasp proposals comprises one or more of a grasp location on the object, grasp pose for an end effector of the robot, a pre-grasp pose of the robot, a post-grasp of the robot, a trajectory for grasping the object, or a grasp force.
3. The method of claim 1 , wherein the one or more criteria includes a function for determining a likelihood of successfully grasping the object using a corresponding candidate grasp proposal.
4. The method of claim 1 , wherein the set of data is determined by a trained neural network, wherein the trained neural network is configured to output pairs of keypoints and corresponding object component identifiers of an object in a scene.
5. The method of claim 1 , wherein an object component identifier of the set of data represents a face of the object.
6. The method of claim 1 , wherein storing the final grasp proposal in memory comprises: adding the final grasp proposal to a data structure.
7. The method of claim 1 , wherein determining the set of candidate grasp locations comprises: calculating, as a candidate grasp location for the object, a center of multiple keypoints each paired with a common object component identifier in the set of data.
8. A system comprising:
one or more computers and one or more non-transitory storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
determining a set of data for an object in a scene, each of the set of data comprising a respective keypoint and a corresponding object component identifier;
determining a set of candidate grasp locations for the object, using the set of data, wherein each candidate grasp location of the set of candidate grasp locations is associated with a respective occlusion score determined by a machine learning model trained on images labeled with predetermined occlusion scores for locations in the images;
determining one or more candidate grasp proposals based on the set of candidate grasp locations for grasping the object;
selecting a candidate grasp proposal from the one or more candidate grasp proposals as a final grasp proposal based on one or more criteria; and
storing the final grasp proposal in memory to be retrieved for controlling a robot to grasp the object.
9. The system of claim 8 , wherein one of the one or more candidate grasp proposals comprises one or more of a grasp location on the object, grasp pose for an end effector of the robot, a pre-grasp pose of the robot, a post-grasp of the robot, a trajectory for grasping the object, or a grasp force.
10. The system of claim 8 , wherein the one or more criteria includes a function for determining a likelihood of successfully grasping the object using a corresponding candidate grasp proposal.
11. The system of claim 8 , wherein the set of data is determined by a trained neural network, wherein the trained neural network is configured to output pairs of keypoints and corresponding object component identifiers of an object in a scene.
12. The system of claim 8 , wherein an object component identifier of the set of data represents a face of the object.
13. The system of claim 8 , wherein storing the final grasp proposal in memory comprises: adding the final grasp proposal to a data structure.
14. The system of claim 8 , wherein determining the set of candidate grasp locations comprises: calculating, as a candidate grasp location for the object, a center of multiple keypoints each paired with a common object component identifier in the set of data.
15. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
determining a set of data for an object in a scene, each of the set of data comprising a respective keypoint and a corresponding object component identifier;
determining a set of candidate grasp locations for the object, using the set of data, wherein each candidate grasp location of the set of candidate grasp locations is associated with a respective occlusion score determined by a machine learning model trained on images labeled with predetermined occlusion scores for locations in the images;
determining one or more candidate grasp proposals based on the set of candidate grasp locations for grasping the object;
selecting a candidate grasp proposal from the one or more candidate grasp proposals as a final grasp proposal based on one or more criteria; and
storing the final grasp proposal in memory to be retrieved for controlling a robot to grasp the object.
16. The one or more non-transitory computer storage media of claim 15 , wherein one of the one or more candidate grasp proposals comprises one or more of a grasp location on the object, grasp pose for an end effector of the robot, a pre-grasp pose of the robot, a post-grasp of the robot, a trajectory for grasping the object, or a grasp force.
17. The one or more non-transitory computer storage media of claim 15 , wherein the one or more criteria includes a function for determining a likelihood of successfully grasping the object using a corresponding candidate grasp proposal.
18. The one or more non-transitory computer storage media of claim 15 , wherein the set of data is determined by a trained neural network, wherein the trained neural network is configured to output pairs of keypoints and corresponding object component identifiers of an object in a scene.
19. The one or more non-transitory computer storage media of claim 15 , wherein an object component identifier of the set of data represents a face of the object.
20. The one or more non-transitory computer storage media of claim 15 , wherein determining the set of candidate grasp locations comprises: calculating, as a candidate grasp location for the object, a center of multiple keypoints each paired with a common object component identifier in the set of data.