Instance-wise, multi-modal, grasp generation systems and methods for robotic end of arm tools

1 . An item grasp generation system, comprising:

a robotic arm having an associated end of arm tool;

an imaging device;

a depth sensor; and

a control system configured to at least:

receive, from the imaging device, imaging data associated with at least one item;

receive, from the depth sensor, depth data associated with the at least one item;

process the imaging data and the depth data to generate at least one of an instance point cloud or instance voxels by at least:

performing instance segmentation upon the imaging data to isolate the at least one item and generate an instance segmentation mask;

overlaying and cropping the imaging data onto the depth data to generate cropped imaging and depth data; and

overlaying the instance segmentation mask onto the cropped imaging and depth data to generate a cropped point cloud with instance mask;

wherein the at least one of the instance point cloud or the instance voxels comprises at least one of the cropped imaging and depth data or the cropped point cloud with instance mask;

at least one of:

process the instance point cloud using a heuristic module; or

process the instance voxels using a learning module;

generate a grasp mode, a grasp position, and a grasp orientation for picking the at least one item based on the processing by at least one of the heuristic module or the learning module; and

instruct the robotic arm having the associated end of arm tool to pick the at least one item using the grasp mode, the grasp position, and the grasp orientation;

wherein the grasp mode comprises at least one of a flat mode, a full pinch mode, a parallel pinch mode, or a small cup mode.

2 . The item grasp generation system of claim 1 , wherein the imaging device is configured to capture imaging data comprising red-green-blue (RGB) imaging data associated with the at least one item.

3 . The item grasp generation system of claim 1 , wherein the depth sensor is configured to capture point cloud depth data associated with the at least one item.

4 . The item grasp generation system of claim 1 , wherein the end of arm tool comprises at least one of a grasping finger, a pinching portion, or a suction cup.

5 . A method, comprising:

receiving, by a control system from an imaging device, imaging data associated with at least one item;

receiving, by the control system from a depth sensor, depth data associated with at least one item;

processing, by the control system, the imaging data and the depth data to generate instance data associated with a single item by at least:

performing instance segmentation upon the imaging data to isolate the single item and generate an instance segmentation mask;

overlaying and cropping the imaging data onto the depth data to generate cropped imaging and depth data; and

overlaying the instance segmentation mask onto the cropped imaging and depth data to generate a cropped point cloud with instance mask;

wherein the instance data comprises at least one of the cropped imaging and depth data or the cropped point cloud with instance mask;

processing, by the control system, the instance data using at least one of a heuristic module or a learning module;

generating, by the control system, a grasp mode, a grasp position, and a grasp orientation for picking the single item based on the processing using at least one of the heuristic module or the learning module; and

instructing, by the control system, a robotic arm having an associated end of arm tool to pick the single item using the grasp mode, the grasp position, and the grasp orientation;

wherein the grasp mode comprises at least one of a flat mode, a full pinch mode, a parallel pinch mode, or a small cup mode.

6 . The method of claim 5 , wherein processing the imaging data and the depth data further comprises:

overlaying the instance segmentation mask onto the depth data to generate an isolated instance point cloud;

wherein the instance data comprises the isolated instance point cloud.

7 . The method of claim 5 , wherein processing the instance data using the heuristic module further comprises:

performing plane segmentation upon the instance data to generate one or more planes associated with the single item; and

for individual planes of the one or more planes:

determining a plane point cloud associated with the individual plane;

fitting a plane to the plane point cloud;

applying a minimum volume bounding box around the fitted plane; and

determining an axis normal to the fitted plane.

8 . The method of claim 7 , wherein processing the instance data using the heuristic module further comprises:

determining that the axis substantially aligns with a z-axis of a global reference frame;

in response to determining that the axis substantially aligns with the z-axis:

determining x-y dimensions of the fitted plane within the minimum volume bounding box;

selecting the grasp mode of a flat mode, a full pinch mode, or a small cup mode based on the x-y dimensions;

selecting the grasp position at a center of the minimum volume bounding box; and

selecting the grasp orientation based on the x-y dimensions.

9 . The method of claim 7 , wherein processing the instance data using the heuristic module further comprises:

determining that the axis does not substantially align with a z-axis of a global reference frame;

in response to determining that the axis does not substantially align with the z-axis:

determining an intersection line between the fitted plane and an upper bounding plane of the minimum volume bounding box;

selecting the grasp mode of a parallel pinch mode;

selecting the grasp position at a center of the intersection line; and

selecting the grasp orientation based on a direction of the intersection line.

10 . The method of claim 5 , wherein processing the imaging data and the depth data further comprises:

generating truncated signed distance function (TSDF) voxels based on the instance data;

generating red-green-blue (RGB) voxels based on the instance data; and

generating instance mask voxels based on the instance data.

11 . The method of claim 10 , wherein processing the imaging data and the depth data further comprises:

encoding the TSDF voxels, the RGB voxels, and the instance mask voxels for the single item as input to the learning module.

12 . The method of claim 5 , wherein the learning module comprises a mode classifier, a position decoder, and an orientation decoder.

13 . The method of claim 12 , wherein processing the instance data using the learning module further comprises:

generating, by the mode classifier, a probability distribution for a plurality of grasp modes;

generating, by the position decoder, a plurality of position maps corresponding to respective grasp modes of the plurality of grasp modes; and

generating, by the orientation decoder, an orientation map corresponding to at least one grasp mode of the plurality of grasp modes.

14 . The method of claim 13 , wherein generating, by the control system, the grasp mode, the grasp position, and the grasp orientation for picking the single item further comprises:

selecting the grasp mode having a highest probability distribution among the plurality of grasp modes;

selecting the grasp position based on a respective position map corresponding to the selected grasp mode; and

selecting the grasp orientation based on the orientation map corresponding to the selected grasp mode.

15 . A method, comprising:

receiving, by a control system from an imaging device, imaging data associated with at least one item;

receiving, by the control system from a depth sensor, depth data associated with at least one item;

processing, by the control system, the imaging data and the depth data to generate instance data associated with a single item by at least:

performing instance segmentation upon the imaging data to isolate the single item and generate an instance segmentation mask;

overlaying and cropping the imaging data onto the depth data to generate cropped imaging and depth data; and

overlaying the instance segmentation mask onto the cropped imaging and depth data to generate a cropped point cloud with instance mask;

wherein the instance data comprises at least one of the cropped imaging and depth data or the cropped point cloud with instance mask;

processing, by the control system, the instance data using a learning module;

generating, by the control system, a first subset including at least one of a grasp mode, a grasp position, or a grasp orientation for picking the single item based on the processing using the learning module;

processing, by the control system, the instance data using a heuristic module based on the first subset;

generating, by the control system, a second subset including at least one of the grasp mode, the grasp position, or the grasp orientation for picking the single item based on the processing using the heuristic module and the first subset; and

instructing, by the control system, a robotic arm having an associated end of arm tool to pick the single item using the grasp mode, the grasp position, and the grasp orientation;

wherein the grasp mode comprises at least one of a flat mode, a full pinch mode, a parallel pinch mode, or a small cup mode.

16 . The method of claim 15 , wherein the first subset generated using the learning module comprises the grasp mode for picking the single item.

17 . The method of claim 15 , wherein the second subset generated using the heuristic module comprises the grasp position and the grasp orientation for picking the single item.