Grasping of an object by a robot based on grasp strategy determined using machine learning model(s)

1. A method implemented by one or more processors,

comprising:

applying vision data as input to at least one trained machine learning model, the vision data being generated by one or more vision components of a robot and capturing features of an object in an environment of the robot and additional features of an alternative object in the environment;

processing the vision data using the at least one trained machine learning model to generate output, of the machine learning model, defining:

a spatial region for interacting with the object to grasp the object, the spatial region being a bounding area that encompasses a portion of the vision data corresponding to the object, and

a semantic indication for the spatial region,

an alternative spatial region for interacting with the alternative object, the alternative spatial region being an alternative bounding area that encompasses an alternative portion of the vision data corresponding to the alternative object, and

an alternative semantic indication for the additional spatial region;

selecting the spatial region based on the spatial region corresponding to the object and the object being selected for grasping;

selecting, based on the semantic indication, a particular grasp strategy of a plurality of candidate grasp strategies, wherein selecting the particular grasp strategy based on the semantic indication is based on the semantic indication being for the spatial region and the spatial region being selected;

determining an end effector pose for interacting with the object to grasp the object, wherein determining the end effector pose is based on the spatial region defined by the output and is based on the particular grasp strategy selected based on the semantic indication defined by the output; and

providing, to actuators of the robot, commands that cause an end effector of the robot to traverse to the end effector pose in association with attempting a grasp of the object.

2. The method of claim 1 , wherein the particular grasp strategy comprises a grasp approach direction for approaching the object in attempting the grasp of the object, and wherein determining the end effector pose is based on the grasp approach direction.

3. The method of claim 2 , wherein the particular grasp strategy further comprises an initial manipulation to perform on the object, prior to attempting the grasp of the object, and further comprising:

providing, to the actuators of the robot, further commands that cause the end effector of the robot to perform the initial manipulation on the object in association with attempting the grasp of the object.

4. The method of claim 3 , wherein the initial manipulation comprises sliding the object across a surface on which the object rests in the environment.

5. The method of claim 1 , wherein the particular grasp strategy comprises at least one of: a degree of force to apply in attempting the grasp of the object, and a grasp type to be performed by the end effector.

6. The method of claim 1 , wherein the output is generated over a single model of the at least one trained machine learning model, and defines the spatial region, defines the semantic indication for the spatial region, defines the alternative spatial region, and defines the alternative semantic indication for the alternative spatial region.

7. The method of claim 6 , wherein the vision data lacks a depth channel.

8. The method of claim 7 , wherein determining the end effector pose comprises:

selecting at least one particular three-dimensional (3D) point, from a group of 3D points, based on the particular 3D point being within the spatial region,

wherein the group of 3D points includes a depth channel, and wherein the group of 3D points is generated by the vision component, or is generated by an additional vision component of the robot that is viewing the environment; and

determining the at least one end effector pose based on the at least one particular 3D point.

9. The method of claim 8 , wherein the group of 3D points is generated by the vision component, and wherein the vision data processed using the single model comprises the group of 3D points without the depth channel.

10. The method of claim 8 , wherein determining the end effector pose based on the at least one particular 3D point comprises determining an approach vector based on a surface normal determined based on the at least one particular 3D point, and determining the end effector pose based on the surface normal.

11. The method of claim 10 , wherein selecting the at least one particular 3D point is further based on the surface normal conforming to a grasp approach direction of the grasp strategy.

12. The method of claim 1 , wherein the semantic indication associated with the object that is defined by the output comprises a classification of the object, and wherein selecting the particular grasp strategy is based on the particular grasp strategy being stored in association with the classification of the object.

13. The method of claim 12 , wherein the output generated by processing the vision data using the at least one trained machine learning model further comprises an additional classification associated with an additional object in the environment, and wherein selecting the particular grasp strategy is based on the particular grasp strategy being stored in association with both: the classification of the object and the additional classification of the additional object.

14. A method implemented by one or more processors, comprising:

applying vision data as input to a trained object classification machine learning model, the vision data being generated by a vision component of a robot and capturing an object in an environment of the robot and an additional object in the environment of the robot;

processing the vision data using the trained object classification machine learning model, to generate output indicating a semantic classification of the object and an additional semantic classification of the additional object;

selecting, from a plurality of candidate grasp strategies and based on both the semantic classification of the object and the additional semantic classification of the additional object, a particular grasp strategy for grasping the object; and

controlling an end effector of the robot to cause the end effector to manipulate the object in accordance with the particular grasp strategy selected based on both the semantic classification of the object and the additional semantic classification of the additional object, in attempting a grasp of the object.

15. The method of claim 14 , wherein selecting the particular grasp strategy based on the semantic classification and the additional semantic classification comprises:

applying the semantic classification and the additional semantic classification as additional input to an additional trained machine learning model;

processing the input using the additional machine learning model to generate additional output that indicates the grasp strategy;

selecting the grasp strategy based on it being indicated by the additional output.

16. The method of claim 15 , wherein the additional output comprises a probability of successful grasp for the grasp strategy and additional probabilities of successful grasp for additional grasp strategies of the candidate grasp strategies, and wherein the additional output indicates the grasp strategy based on the probability for the grasp strategy satisfying a threshold.