SUPERVISED AUTONOMOUS GRASPING

1 . A computer-implemented method when executed by data processing hardware of a robot causes the data processing hardware to perform operations comprising:

receiving a user input selecting a target object for grasping by an end-effector of the robot;

receiving initial sensor data including information about the target object;

determining an initial grasp geometry for the end effector to grasp the target object;

instructing the end-effector to grasp the target object based on the initial grasp geometry;

receiving updated sensor data in response to the end-effector executing the initial grasp geometry, the updated sensor data including additional information about the target object indicating that a foreign object is present that obstructs the end-effector from grasping the target object based on the initial grasp geometry, the foreign object not included in the initial sensor data;

modifying the initial grasp geometry based on the additional information about the target object such that the foreign object does not obstruct the end-effector from grasping the target object based on the modified initial grasp geometry; and

controlling the end-effector to grasp the target object based on the modified initial grasp geometry.

2 . The method of claim 1 , wherein determining the initial grasp geometry comprises:

generating a plurality of candidate grasp geometries based on the target object;

for each candidate grasp geometry of the plurality of candidate grasp geometries, determining a grasping score for grasping the target object, the grasping score indicating a likelihood of success for grasping the target object using the end-effector based on the respective candidate grasp geometry; and

selecting the respective candidate grasp geometry with a greatest grasping score as the initial grasp geometry designated to grasp the target object.

3 . The method of claim 2 , wherein the operations further comprise:

determining a new set of candidate grasping geometries based on the updated sensor data;

for each candidate grasping geometry of the new set of candidate grasping geometries, determining a grasping score for grasping the target object; and

determining that the grasping score of a first one of the candidate grasp geometries from the new set of candidate grasping geometries exceeds the grasping score of the initial grasp geometry,

wherein modifying the initial grasp geometry is further based on the first candidate grasp geometry from the new set of candidate grasping geometries.

4 . The method of claim 1 , wherein the operations further comprise receiving an end-effector constraint constraining one or more degrees of freedom for the end-effector of the robotic manipulator to grasp the target object.

5 . The method of claim 1 , wherein instructing the end-effector to grasp the target object based on the modified initial grasp geometry comprises:

instructing a body of the robot to pitch toward the target object; or

instructing a first leg of the robot to rotate an upper member of the first leg about a knee joint towards a lower member of the first leg.

6 . The method of claim 1 , wherein receiving the user input occurs at a user device in remote communication with the data processing hardware of the robot.

7 . The method of claim 1 , wherein the end-effector comprises a gripper having a movable jaw and a fixed jaw, the movable jaw configured to move relative to the fixed jaw to move between an open position for the gripper and a closed position for the gripper.

8 . A robot comprising:

a body;

a plurality of legs coupled to the body;

an end-effector configured to grasp objects within an environment about the robot;

data processing hardware in communication with the end-effector; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving a user input selecting a target object for grasping by an end-effector of the robot;

receiving initial sensor data including information about the target object;

determining an initial grasp geometry for the end effector to grasp the target object;

instructing the end-effector to grasp the target object based on the initial grasp geometry;

receiving updated sensor data in response to the end-effector executing the initial grasp geometry, the updated sensor data including additional information about the target object indicating that a foreign object is present that obstructs the end-effector from grasping the target object based on the initial grasp geometry, the foreign object not included in the initial sensor data;

modifying the initial grasp geometry based on the additional information about the target object such that the foreign object does not obstruct the end-effector from grasping the target object based on the modified initial grasp geometry; and

controlling the end-effector to grasp the target object based on the modified initial grasp geometry.

9 . The robot of claim 8 , wherein determining the initial grasp geometry for the robotic manipulator to grasp the target object within the grasp region comprises:

generating a plurality of candidate grasp geometries based on the target object;

for each candidate grasp geometry of the plurality of candidate grasp geometries, determining a grasping score for grasping the target object, the grasping score indicating a likelihood of success for grasping the target object using the end-effector based on the respective candidate grasp geometry; and

selecting the respective candidate grasp geometry with a greatest grasping score as the initial grasp geometry designated to grasp the target object.

10 . The robot of claim 9 , wherein the operations further comprise:

determining a new set of candidate grasping geometries based on the updated sensor data;

for each candidate grasping geometry of the new set of candidate grasping geometries, determining a grasping score for grasping the target object; and

determining that the grasping score of a first one of the candidate grasp geometries from the new set of candidate grasping geometries exceeds the grasping score of the initial grasp geometry,

wherein modifying the initial grasp geometry is further based on the first candidate grasp geometry from the new set of candidate grasping geometries.

11 . The robot of claim 8 , wherein the operations further comprise receiving an end-effector constraint constraining one or more degrees of freedom for the end-effector of the robotic manipulator to grasp the target object.

12 . The robot of claim 8 , wherein instructing the end-effector to grasp the target object within the grasp region based on the modified initial grasp geometry comprises:

instructing the body of the robot to pitch toward the target object; or

instructing a first leg of the plurality of legs of the robot to rotate an upper member of the first leg about a knee joint towards a lower member of the first leg.

13 . The robot of claim 8 , wherein receiving the selection input occurs at a user device in remote communication with the data processing hardware of the robot.

14 . The robot of claim 8 , wherein the end-effector comprises a gripper having a movable jaw and a fixed jaw, the movable jaw configured to move relative to the fixed jaw to move between an open position for the gripper and a closed position for the gripper.

15 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by a data processing hardware, cause the data processing hardware to:

receive a user input selecting a target object for grasping by an end-effector of the robot;

receiving initial sensor data including information about the target object;

determine an initial grasp geometry for the end effector to grasp the target object;

instruct the end-effector to grasp the target object based on the initial grasp geometry;

receive updated sensor data in response to the end-effector executing the initial grasp geometry, the updated sensor data including additional information about the target object indicating that a foreign object is present that obstructs the end-effector from grasping the target object based on the initial grasp geometry, the foreign object not included in the initial sensor data;

modify the initial grasp geometry based on the additional information about the target object such that the foreign object does not obstruct the end-effector from grasping the target object based on the modified initial grasp geometry; and

control the end-effector to grasp the target object based on the modified initial grasp geometry.

16 . The non-transitory computer-readable medium of claim 15 , wherein determining the initial grasp geometry comprises:

generating a plurality of candidate grasp geometries based on the target object;

for each candidate grasp geometry of the plurality of candidate grasp geometries, determining a grasping score for grasping the target object, the grasping score indicating a likelihood of success for grasping the target object using the end-effector based on the respective candidate grasp geometry; and

selecting the respective candidate grasp geometry with a greatest grasping score as the initial grasp geometry designated to grasp the target object.

17 . The non-transitory computer-readable medium of claim 16 , wherein the instructions, when executed by the data processing hardware, further cause the data processing hardware to:

determine a new set of candidate grasping geometries based on the updated sensor data;

for each candidate grasping geometry of the new set of candidate grasping geometries, determine a grasping score for grasping the target object; and

determine that the grasping score of a first one of the candidate grasp geometries from the new set of candidate grasping geometries exceeds the grasping score of the initial grasp geometry,

wherein modifying the initial grasp geometry is further based on the first candidate grasp geometry from the new set of candidate grasping geometries.

18 . The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the data processing hardware, further cause the data processing hardware to receiving an end-effector constraint constraining one or more degrees of freedom for the end-effector of the robotic manipulator to grasp the target object.

19 . The non-transitory computer-readable medium of claim 15 , wherein instructing the end-effector to grasp the target object based on the modified initial grasp geometry comprises:

instructing a body of the robot to pitch toward the target object; or

instructing a first leg of the robot to rotate an upper member of the first leg about a knee joint towards a lower member of the first leg.

20 . The non-transitory computer-readable medium of claim 15 , wherein receiving the user input occurs at a user device in remote communication with the data processing hardware of the robot.