Geometry-aware instance segmentation in stereo image capture processes

1. A method of detecting multiple instances of an object in a digital image, the method comprising, by a processor:

receiving a two-dimensional (2D) image that includes a plurality of instances of an object in an environment;

using a object detection network to generate a plurality of predicted instances of the object in the 2D image;

generating a data set that comprises depth information by receiving a stereo image pair captured by a stereo camera and estimating a disparity map from the stereo image pair;

accessing the data set that comprises depth information corresponding to the plurality of instances of the object in the environment; and

using the depth information to identify an individual instance from the plurality of predicted instances of the object in the image by:

projecting the predicted instances of the object in the 2D image to the disparity map and removing a plurality of predicted 2.5D masks from the disparity map to yield a cropped disparity map,

back-projecting the removed candidate masks to a three-dimensional (3D) coordinate space to yield a plurality of predicted 3D masks,

generating a first score for each of the predicted instances of the object in the 2D image,

generating a second score for each of the predicted 2.5D masks,

generating a third score for each of the predicted 3D masks, and

using the first scores, the second scores and the third scores to predict a final mask.

2. The method of claim 1 , wherein:

receiving the 2D image comprises receiving the 2D image from a first sensing modality of an autonomous vehicle (AV); and

receiving the data set comprises receiving the data set from a second sensing modality of the AV.

3. The method of claim 2 , wherein

the second sensing modality comprises a stereo camera; and

the data set that comprises depth information comprises a disparity map that is generated from an image pair captured by the stereo camera.

4. The method of claim 3 , wherein receiving the 2D image comprises receiving one image from the image pair captured by the stereo camera.

5. The method of claim 1 , wherein using the object detection network to generate the plurality of predicted instances of the object in the image comprises using a region proposal network to output a plurality of bounding box proposals.

6. The method of claim 1 , wherein using the first scores, the second scores and the third scores to predict the final mask comprises:

fusing the second scores and the third scores to generate a disparity mask score for each of a plurality of candidate final masks;

fusing the disparity mask scores with the first scores to generate plurality of predicted final mask scores; and

using the final mask scores to select the final mask.

7. A system for detecting multiple instances of an object in an image, the system comprising:

a processor; and

a computer-readable memory containing programming instructions that are configured to cause the processor to:

receive a two-dimensional (2D) image that includes a plurality of instances of an object in an environment;

use a object detection network to generate a plurality of predicted instances of the object in the 2D image;

receive a stereo image pair captured by a stereo camera and estimate a disparity map from the stereo image pair to generate a data set that comprises depth information; and

access the data set that comprises depth information corresponding to the plurality of instances of the object in the environment; and

use the depth information to identify an individual instance from the plurality of predicted instances of the object in the image via additional programming instructions configured to further cause the processor to:

project the predicted instances of the object in the 2D image to the disparity map and remove a plurality of predicted 2.5D masks from the disparity map to yield a cropped disparity map,

back-project the removed candidate masks to a three-dimensional (3D) coordinate space to yield a plurality of predicted 3D masks,

generate a first score for each of the predicted instances of the object in the 2D image,

generate a second score for each of the predicted 2.5D masks,

generate a third score for each of the predicted 3D masks, and

use the first scores, the second scores and the third scores to predict a final mask.

8. The system of claim 7 , wherein:

the instructions to receive the 2D image comprise instructions to receive the 2D image from a first sensing modality of an autonomous vehicle (AV); and

the instructions to receive the data set comprise instructions to receive the data set from a second sensing modality of the AV.

9. The system of claim 8 , wherein

the second sensing modality comprises a stereo camera of the AV; and

the data set that comprises depth information comprises a disparity map that is to be generated from an image pair captured by the stereo camera.

10. The system of claim 9 , wherein the instructions to receive the 2D image comprise instructions to receive one image from the image pair captured by the stereo camera.

11. The system of claim 7 , wherein the instructions to use the object detection network to generate the plurality of predicted instances of the object in the image comprise instructions to use a region proposal network to output a plurality of bounding box proposals.

12. The system of claim 7 , wherein the instructions to use the first scores, the second scores and the third scores to predict the final mask comprise instructions to:

fuse the second scores and the third scores to generate a disparity mask score for each of a plurality of candidate final masks;

fuse the disparity mask scores with the first scores to generate plurality of predicted final mask scores; and

use the final mask scores to select the final mask.

13. The system of claim 8 , wherein:

the processor is a component of the AV;

the first sensing modality comprises either a right side or a left side of a stereo camera of the AV; and

the second sensing modality comprises the right side and the left side of the stereo camera.

14. A computer program embodied in a memory device, the computer program comprising programming instructions that are configured to cause a processor to:

receive a two-dimensional (2D) image that includes a plurality of instances of an object in an environment;

use a object detection network to generate a plurality of predicted instances of the object in the 2D image;

receive a stereo image pair captured by a stereo camera and estimate a disparity map from the stereo image pair to generate a data set that comprises depth information;

access the data set that comprises depth information corresponding to the plurality of instances of the object in the environment; and

use the depth information to identify an individual instance from the plurality of predicted instances of the object in the image via additional programming instructions configured to further cause the processor to:

project the predicted instances of the object in the 2D image to the disparity map and remove a plurality of predicted 2.5D masks from the disparity map to yield a cropped disparity map,

back-project the removed candidate masks to a three-dimensional (3D) coordinate space to yield a plurality of predicted 3D masks,

generate a first score for each of the predicted instances of the object in the 2D image,

generate a second score for each of the predicted 2.5D masks,

generate a third score for each of the predicted 3D masks, and

use the first scores, the second scores and the third scores to predict a final mask.

15. The computer program of claim 14 , wherein:

the instructions to receive the 2D image comprise instructions to receive the 2D image from a first sensing modality of an autonomous vehicle (AV); and

the instructions to receive the data set comprise instructions to receive the data set from a second sensing modality of the AV.

16. The computer program of claim 15 , wherein

the second sensing modality comprises a stereo camera; and

the data set that comprises depth information comprises a disparity map that is to be generated from an image pair captured by the stereo camera.

17. The computer program of claim 16 , wherein the instructions to receive the 2D image comprise instructions to receive one image from the image pair captured by the stereo camera.

18. The computer program of claim 15 , wherein the instructions to use the object detection network to generate the plurality of predicted instances of the object in the image comprise instructions to use a region proposal network to output a plurality of bounding box proposals.

19. The system of claim 14 , wherein the instructions to use the first scores, the second scores and the third scores to predict the final mask comprise instructions to:

fuse the second scores and the third scores to generate a disparity mask score for each of a plurality of candidate final masks;

fuse the disparity mask scores with the first scores to generate plurality of predicted final mask scores; and

use the final mask scores to select the final mask.