Robotic control based on 3D bounding shape, for an object, generated using edge-depth values for the object

1. A method implemented by one or more processors of a robot, the method comprising:

capturing a first infrared image that captures, from a first point of view, an infrared projection projected onto one or more objects in an environment of the robot, the infrared projection projected by an infrared projector of the robot;

capturing a second infrared image that captures, from a second point of view, the infrared projection projected onto the one or more objects;

generating an edge-depth image that includes first corresponding edge-depth values for only detected edges in the first infrared image and the second infrared image, wherein generating the edge-depth image comprises:

performing local contrast normalization on the first infrared image to determine first edge pixels, of the first infrared image, that constitute edges,

performing local contrast normalization on the second infrared image to determine second edge pixels, of the second infrared image, that constitute edges, and

processing only the first edge pixels and the second edge pixels to generate the edge-depth image;

generating a 3D point cloud for a given object of the one or more objects, wherein generating the 3D point cloud is based on the corresponding edge-depth values determined to correspond to the given object;

generating a 3D bounding shape for the given object, wherein generating the 3D bounding shape comprises processing the 3D point cloud using a 3D bounding shape prediction model; and

controlling one or more actuators of the robot based on the 3D bounding shape.

2. The method of claim 1 , further comprising:

capturing a red, green, blue (RGB) image that captures the one or more objects.

3. The method of claim 2 , further comprising:

processing the RGB image using an object detection model to generate a two-dimensional (2D) bounding shape;

wherein generating the 3D point cloud for the given object comprises:

determining the edge-depth values that correspond to the given object based on the edge-depth values being within a frustum generated based on RGB pixels, of the RGB image, that are within the 2D bounding shape.

4. The method of claim 3 , wherein generating the 3D point cloud for the given object further comprises:

generating candidate 3D points for the 3D point cloud based on the candidate 3D points having corresponding depth values within the frustum, the candidate 3D points including edge-depth 3D points generated based on the first corresponding edge-depth values and the second corresponding edge-depth values; and

generating the 3D point cloud for the given object based on processing the candidate 3D points using an object segmentation model.

5. The method of claim 1 , further comprising:

generating a regular depth image that includes corresponding regular depth values, the regular depth values including depth values that are in addition to the edge-depth values; and

wherein generating the 3D point cloud is further based on the regular depth values determined to correspond to the given object.

6. The method of claim 5 , wherein the given object comprises at least one transparent surface, and wherein a majority of the regular depth values fail to define any depth values that correspond to the at least one transparent surface.

7. The method of claim 1 , wherein the first infrared image is captured by a first infrared sensor of the robot and the second infrared image is captured by a second infrared sensor of the robot.

8. The method of claim 1 , wherein the first infrared image is captured by an infrared sensor of the robot at the first point of view at a first time, and the second infrared image is captured by the infrared sensor of the robot at the second point of view at a second time.

9. The method of claim 1 , wherein controlling the one or more actuators of the robot based on the 3D bounding shape comprises:

controlling the one or more actuators, based on the 3D bounding shape, to manipulate the given object.

10. The method of claim 1 , wherein the given object comprises at least one transparent surface.

11. The method of claim 1 , further comprising:

capturing a red, green, blue (RGB) image that captures the one or more objects;

processing the RGB image using an object classification model to generate a predicted classification of the given object;

wherein generating the 3D point cloud for the given object is further based on the predicted classification.

12. A method implemented by one or more processors of a robot, the method comprising:

capturing a first image that captures, from a first point of view, one or more objects in an environment of the robot;

capturing a second image that captures the one or more objects from a second point of view;

generating an edge-depth image that includes first corresponding edge-depth values for only detected edges in the first image and the second image, wherein generating the edge-depth image comprises:

performing local contrast normalization on the first image to determine first edge pixels, of the first image, that constitute edges,

performing local contrast normalization on the second image to determine second edge pixels, of the second image, that constitute edges, and

processing only the first edge pixels and the second edge pixels to generate the edge-depth image;

generating a 3D point cloud for a given object of the one or more objects, wherein generating the 3D point cloud is based on the corresponding edge-depth values determined to correspond to the given object;

generating a 3D bounding shape for the given object, wherein generating the 3D bounding shape comprises processing the 3D point cloud using a three-dimensional (3D) bounding shape prediction model; and

controlling one or more actuators of the robot based on the 3D bounding shape.

13. The method of claim 12 , further comprising:

generating a regular depth image that includes corresponding regular depth values, the regular depth values including depth values that are in addition to the edge-depth values; and

wherein generating the 3D point cloud is further based on the regular depth values determined to correspond to the given object.

14. The method of claim 13 , wherein the given object comprises at least one transparent surface, and wherein a majority of the regular depth values fail to define any depth values that correspond to the at least one transparent surface.

15. The method of claim 12 , wherein the first image is captured by a first sensor of the robot and the second image is captured by a second sensor of the robot.

16. The method of claim 15 , wherein the first sensor is a first color sensor and the second sensor is a second color sensor.

17. A robot comprising:

actuators;

an infrared projector that projects an infrared projection onto one or more objects in an environment of the robot;

a first infrared sensor that captures a first infrared image that captures, from a first point of view, the infrared projection projected onto the one or more objects;

a second infrared sensor that captures a second infrared image that captures, from a second point of view, the infrared projection projected onto the one or more objects; and

one or more processors executing stored instructions to:

generate an edge-depth image that includes first corresponding edge-depth values for only detected edges in the first infrared image and the second infrared image, wherein in generating the edge-depth image one or more of the processors are to:

perform local contrast normalization on the first infrared image to determine first edge pixels, of the first infrared image, that constitute edges,

perform local contrast normalization on the second infrared image to determine second edge pixels, of the second infrared image, that constitute edges, and

process only the first edge pixels and the second edge pixels to generate the edge-depth image;

generate a 3D point cloud for a given object of the one or more objects, wherein generating the 3D point cloud is based on the corresponding edge-depth values determined to correspond to the given object;

generate a 3D bounding shape for the given object, wherein generating the 3D bounding shape comprises processing the 3D point cloud using a 3D bounding shape prediction model; and

control the actuators of the robot based on the 3D bounding shape.

18. The robot of claim 17 , further comprising:

a red, green, blue (RGB) sensor that captures an RGB image that captures the one or more objects.

19. The robot of claim 18 , wherein the one or more processors, in executing the stored instructions, are further to:

process the RGB image using an object detection model to generate a two-dimensional (2D) bounding shape;

wherein in generating the 3D point cloud for the given object one or more of the processors are to:

determine the edge-depth values that correspond to the given object based on the edge-depth values being within a frustum generated based on RGB pixels, of the RGB image, that are within the 2D bounding shape.

20. The robot of claim 19 , wherein in generating the 3D point cloud for the given object one or more of the processors are further to :

generate candidate 3D points for the 3D point cloud based on the candidate 3D points having corresponding depth values within the frustum, the candidate 3D points including edge-depth 3D points generated based on the first corresponding edge-depth values and the second corresponding edge-depth values; and

generate the 3D point cloud for the given object based on processing the candidate 3D points using an object segmentation model.