Geometry-aware instance segmentation in stereo image capture processes
A system detects multiple instances of an object in a digital image by receiving a two-dimensional (2D) image that includes a plurality of instances of an object in an environment. For example, the system may receive the 2D image from a camera or other sensing modality of an autonomous vehicle (AV). The system uses a first object detection network to generate a plurality of predicted object instances in the image. The system then receives a data set that comprises depth information corresponding to the plurality of instances of the object in the environment. The data set may be received, for example, from a stereo camera of an AV, and the depth information may be in the form of a disparity map. The system may use the depth information to identify an individual instance from the plurality of predicted object instances in the image.
1. A method of detecting multiple instances of an object in a digital image, the method comprising, by a processor:
receiving a two-dimensional (2D) image that includes a plurality of instances of the object in an environment;
using an object detection network to generate a plurality of predicted instances of the object in the 2D image;
accessing a data set that comprises depth information corresponding to the plurality of instances of the object in the environment;
projecting the plurality of predicted instances of the object in the 2D image to the depth information corresponding to the plurality of instances of the object in the environment;
removing a plurality of predicted 2.5D masks from the depth information corresponding to the plurality of instances of the object in the environment; and
back-projecting the removed 2.5D masks to a three-dimensional (3D) coordinate space to yield a plurality of predicted 3D masks.
2. The method of claim 1 , wherein using the object detection network to generate the plurality of predicted instances of the object in the image comprises using a region proposal network to output a plurality of bounding box proposals.
3. The method of claim 1 , wherein the data set that comprises depth information comprises a disparity map generated by a stereo matching network.
4. The method of claim 1 , wherein:
receiving the 2D image comprises receiving the 2D image from a first sensing modality of an autonomous vehicle (AV); and
receiving the data set comprises receiving the data set from a second sensing modality of the AV.
5. The method of claim 4 , wherein
the second sensing modality comprises a stereo camera; and
the data set that comprises depth information comprises a disparity map that is generated from an image pair captured by the stereo camera.
6. The method of claim 5 , wherein receiving the 2D image comprises receiving one image from the image pair captured by the stereo camera.
7. A system for detecting multiple instances of an object in a digital image, the system comprising:
a processor and a computer-readable memory containing programming instructions that are configured to cause the processor to:
receive a two-dimensional (2D) image that includes a plurality of instances of the object in an environment;
use an object detection network to generate a plurality of predicted instances of the object in the 2D image;
access a data set that comprises depth information corresponding to the plurality of instances of the object in the environment;
project the plurality of predicted instances of the object in the 2D image to the depth information corresponding to the plurality of instances of the object in the environment;
remove a plurality of predicted 2.5D masks from the depth information corresponding to the plurality of instances of the object in the environment; and
back-project the removed 2.5D masks to a three-dimensional (3D) coordinate space to yield a plurality of predicted 3D masks.
8. The system of claim 7 , wherein the instructions to use the object detection network to generate the plurality of predicted instances of the object in the image comprise instructions to use a region proposal network to output a plurality of bounding box proposals.
9. The method of claim 1 , wherein the data set that comprises depth information comprises a disparity map generated by a stereo matching network.
10. The system of claim 7 , wherein:
the instructions to receive the 2D image comprise instructions to receive the 2D image from a first sensing modality of an autonomous vehicle (AV); and
the instructions to receive the data set comprise instructions to receive the data set from a second sensing modality of the AV.
11. The system of claim 10 , wherein
the second sensing modality comprises a stereo camera of the AV; and
the data set that comprises depth information comprises a disparity map that is to be generated from an image pair captured by the stereo camera.
12. The system of claim 11 , wherein the instructions to receive the 2D image comprise instructions to receive one image from the image pair captured by the stereo camera.
13. A computer program embodied in a memory device, the computer program comprising programming instructions that are configured to cause a processor to:
receive a two-dimensional (2D) image that includes a plurality of instances of an object in an environment;
use an object detection network to generate a plurality of predicted instances of the object in the 2D image;
access a data set that comprises depth information corresponding to the plurality of instances of the object in the environment;
project the plurality of predicted instances of the object in the 2 D image to the depth information corresponding to the plurality of instances of the object in the environment;
remove a plurality of predicted 2.5D masks from the depth information corresponding to the plurality of instances of the object in the environment; and
back-project the removed 2.5D masks to a three-dimensional (3D) coordinate space to yield a plurality of predicted 3D masks.
14. The computer program of claim 13 , wherein the instructions to use the object detection network to generate the plurality of predicted instances of the object in the image comprise instructions to use a region proposal network to output a plurality of bounding box proposals.
15. The computer program of claim 13 , wherein the data set that comprises depth information comprises a disparity map generated by a stereo matching network.
16. The computer program of claim 13 , wherein:
the instructions to receive the 2D image comprise instructions to receive the 2D image from a first sensing modality of an autonomous vehicle (AV); and
the instructions to receive the data set comprise instructions to receive the data set from a second sensing modality of the AV.
17. The computer program of claim 16 , wherein
the second sensing modality comprises a stereo camera of the AV; and
the data set that comprises depth information comprises a disparity map that is to be generated from an image pair captured by the stereo camera.