Synergies between pick and place: task-aware grasp estimation
Systems, methods, and apparatuses for controlling a robot including a manipulator, including: determining three-dimensional (3D) geometry information about a target object based on an image of the target object; determining 3D geometry information about a scene in which the target object is to be placed based on at least one image of the scene; obtaining affordance information by providing the 3D geometry information about the target object and the 3D geometry information about the scene to at least one neural network model; commanding the robot to grasp the target object using the manipulator according to a grasp orientation corresponding to the affordance information; and commanding the robot to position the manipulator according to a placement direction corresponding to the affordance information in order to place the target object at a location in the scene.
1 . An electronic device for controlling a robot including a manipulator, the electronic device comprising:
one or more processors configured to:
determine three-dimensional (3D) geometry information about a target object based on an image of the target object;
determine 3D geometry information about a scene in which the target object is to be placed based on at least one image of the scene;
determine a plurality of candidate placement directions for placing the target object in the scene based on the 3D geometry information about the scene;
obtain affordance information by providing the 3D geometry information about the target object and information about the plurality of candidate placement directions to at least one neural network model;
command the robot to grasp the target object using the manipulator according to a grasp orientation corresponding to the affordance information; and
command the robot to position the manipulator according to a placement direction corresponding to the affordance information in order to place the target object at a location in the scene.
2 . The electronic device of claim 1 , wherein the one or more processors are further configured to:
determine a plurality of candidate grasp orientations for grasping the target object based on the 3D geometry information about the target object;
obtain a plurality of affordance maps by providing information about the plurality of candidate grasp orientations and the information about the plurality of candidate placement directions to the at least one neural network model, wherein each affordance map from among the plurality of affordance maps corresponds to a candidate grasp orientation from among the plurality of candidate grasp orientations and a candidate placement direction from among the plurality of candidate placement directions; and
select an affordance map from among the plurality of affordance maps, wherein the affordance information corresponds to the selected affordance map.
3 . The electronic device of claim 2 , wherein the at least one neural network model comprises:
an object encoder configured to output a plurality of object encodings corresponding to the plurality of candidate grasp orientations based on the 3D geometry information about the target object;
a scene encoder configured to output a plurality of scene encodings corresponding to the plurality of candidate placement directions based on the 3D geometry information about the scene; and
an affordance decoder configured to output the plurality of affordance maps based on the plurality of object encodings and the plurality of scene encodings.
4 . The electronic device of claim 3 , wherein the object encoder, the scene encoder, and the affordance decoder are jointly trained.
5 . The electronic device of claim 2 , wherein the affordance map comprises a plurality of pixels corresponding to a plurality of affordance values, and
wherein each affordance value from among the plurality of affordance values indicates a probability of success for placing the target object.
6 . The electronic device of claim 5 , wherein the affordance map is selected based on the plurality of pixels including a highest affordance value from among all affordance values associated with the plurality of affordance maps.
7 . The electronic device of claim 1 , further comprising at least one camera configured to capture the image of the target object and the at least one image of the scene.
8 . The electronic device of claim 7 , wherein the image of the target object is a depth image, and
wherein the at least one image of the scene is a color image.
9 . The electronic device of claim 1 , wherein the one or more processors are configured to command the robot to position the manipulator by computing a proposed trajectory based on the placement direction, and generating a velocity command corresponding to the proposed trajectory.
10 . A method for controlling a robot including a manipulator, the method comprising:
determining three-dimensional (3D) geometry information about a target object based on an image of the target object;
determining 3D geometry information about a scene in which the target object is to be placed based on at least one image of the scene;
determining a plurality of candidate placement directions for placing the target object in the scene based on the 3D geometry information about the scene;
obtaining affordance information by providing the 3D geometry information about the target object and information about the plurality of candidate placement directions to at least one neural network model;
commanding the robot to grasp the target object using the manipulator according to a grasp orientation corresponding to the affordance information; and
commanding the robot to position the manipulator according to a placement direction corresponding to the affordance information in order to place the target object at a location in the scene.
11 . The method of claim 10 , further comprising:
determining a plurality of candidate grasp orientations for grasping the target object based on the 3D geometry information about the target object;
obtaining a plurality of affordance maps by providing information about the plurality of candidate grasp orientations and the information about the plurality of candidate placement directions to the at least one neural network model, wherein each affordance map from among the plurality of affordance maps corresponds to a candidate grasp orientation from among the plurality of candidate grasp orientations and a candidate placement direction from among the plurality of candidate placement directions; and
selecting an affordance map from among the plurality of affordance maps,
wherein the affordance information corresponds to the selected affordance map.
12 . The method of claim 11 , wherein the at least one neural network model comprises:
an object encoder configured to output a plurality of object encodings corresponding to the plurality of candidate grasp orientations based on the 3D geometry information about the target object;
a scene encoder configured to output a plurality of scene encodings corresponding to the plurality of candidate placement directions based on the 3D geometry information about the scene; and
an affordance decoder configured to output the plurality of affordance maps based on the plurality of object encodings and the plurality of scene encodings.
13 . The method of claim 12 , wherein the object encoder, the scene encoder, and the affordance decoder are jointly trained.
14 . The method of claim 11 , wherein the affordance map comprises a plurality of pixels corresponding to a plurality of affordance values, and
wherein each affordance value from among the plurality of affordance values indicates a probability of success for placing the target object.
15 . The method of claim 14 , wherein the affordance map is selected based on the plurality of pixels including a highest affordance value from among all affordance values associated with the plurality of affordance maps.
16 . The method of claim 10 , further comprising capturing the image of the target object and the at least one image of the scene.
17 . The method of claim 16 , wherein the image of the target object is a depth image, and
wherein the at least one image of the scene is a color image.
18 . The method of claim 10 , wherein the commanding the robot to position the manipulator comprises computing a proposed trajectory based on the placement direction, and generating a velocity command corresponding to the proposed trajectory.
19 . A non-transitory computer-readable medium configured to store instructions which, when executed by at least one processor of a device for controlling a robot including a manipulator, cause the at least one processor to:
determine three-dimensional (3D) geometry information about a target object based on an image of the target object;
determine 3D geometry information about a scene in which the target object is to be placed based on at least one image of the scene;
determine a plurality of candidate placement directions for placing the target object in the scene based on the 3D geometry information about the scene;
obtain affordance information by providing the 3D geometry information about the target object and information about the plurality of candidate placement directions to at least one neural network model;
command the robot to grasp the target object using the manipulator according to a grasp orientation corresponding to the affordance information; and
command the robot to position the manipulator according to a placement direction corresponding to the affordance information in order to place the target object at a location in the scene.
20 . The non-transitory computer-readable medium of claim 19 , the instructions further cause the at least one processor to:
determine a plurality of candidate grasp orientations for grasping the target object based on the 3D geometry information about the target object; and
obtain a plurality of affordance maps by providing information about the plurality of candidate grasp orientations and the information about the plurality of candidate placement directions to the at least one neural network model, wherein each affordance map from among the plurality of affordance maps corresponds to a candidate grasp orientation from among the plurality of candidate grasp orientations and a candidate placement direction from among the plurality of candidate placement directions; and
select an affordance map from among the plurality of affordance maps,
wherein the affordance information corresponds to the selected affordance map.