LiDAR point selection using image segmentation
The subject disclosure relates to techniques for selecting points of an image for processing with LiDAR data. A process of the disclosed technology can include steps for receiving an image comprising a first image object and a second image object, processing the image to place a bounding box around the first image object and the second image object, and processing an image area within the bounding box to identify a first image mask corresponding with a first pixel region of the first image object and a second image mask corresponding with a second pixel region of the second image object. Systems and machine-readable media are also provided.
1. A computer-implemented method, comprising:
receiving, from a first data set recorded by one or more cameras, an image comprising an image object;
processing the image to place a bounding box around the image object;
processing an image area within the bounding box to identify an image mask corresponding with a pixel region of the image object;
processing the image area within the bounding box to identify a second image mask corresponding with a second pixel region of a second image object in the image;
identifying one or more pixels in the first image object and one or more second pixels in the second image object in the pixel region using the bounding box and the image mask; and
processing the one or more pixels to determine a classification for the first and second image object, the classification being used in 3-dimensional transformation of the image for controlling operation of an Autonomous Vehicle (AV).
2. The computer-implemented method of claim 1 , further comprising:
processing the one or more pixels to determine a depth of the image object based on LiDAR data.
3. The computer-implemented method of claim 1 , wherein processing the image to place the bounding box around the image object is performed using a first machine-learning model.
4. The computer-implemented method of claim 1 , wherein processing the image area within the bounding box to identify the image mask is performed using a second machine-learning model.
5. The computer-implemented method of claim 1 , further comprising:
associating a range with the image object based on LiDAR data.
6. A system, comprising:
one or more processors; and
a computer-readable medium comprising instructions stored therein, which when executed by the processors, cause the processors to:
receive, from a first data set recorded by one or more cameras, an image comprising a first image object and a second image object;
process the image to place a bounding box around the first image object and the second image object;
process an image area within the bounding box to identify a first image mask corresponding with a first pixel region of the first image object and a second image mask corresponding with a second pixel region of the second image object;
identify one or more first pixels in the first image object and the second image object using the bounding box, the first image mask, and the second image mask;
processing one or more pixels in the first pixel region to determine a first classification label for the first image object;
processing one or more pixels in the second pixel region to determine a second classification label for the second image object;
process the one or more pixels to determine a classification for the first image object and the second image object, the classification being used in 3-dimensional transformation of the image for controlling operation of an Autonomous Vehicle (AV); and
associating a first classification label with the first image object and a second classification label with the second image object.
7. The system of claim 6 , wherein the one or more processors are configured to execute the computer-readable instructions to:
process the first pixel region to determine a depth of the first image object based on LiDAR data.
8. The system of claim 6 , wherein the one or more processors are configured to execute the computer-readable instructions to:
process the second pixel region to determine a depth of the second image object based on LiDAR data.
9. The system of claim 6 , wherein the one or more processors are configured to execute the computer-readable instructions to process the image to place the bounding box around the first image object and the second image object is performed using a first machine-learning model.
10. The system of claim 6 , wherein the one or more processors are configured to execute the computer-readable instructions to process the image area within the bounding box to identify the first image mask is performed using a second machine-learning model.
11. The system of claim 6 , wherein the one or more processors are configured to execute the computer-readable instructions to:
associate a first range with the first image object and a second range with the second image object based on LiDAR data.
12. A non-transitory computer-readable storage medium comprising instructions stored therein, which when executed by one or more processors, cause the processors to:
receive, from a first data set recorded by one or more cameras, an image comprising a first image object and a second image object;
process the image to place a bounding box around the first image object and the second image object;
process an image area within the bounding box to identify a first image mask corresponding with a first pixel region of the first image object and a second image mask corresponding with a second pixel region of the second image object;
identify one or more first pixels in the first image object and the second image object using the bounding box, the first image mask, and the second image mask;
processing one or more pixels in the first pixel region to determine a first classification label for the first image object;
processing one or more pixels in the second pixel region to determine a second classification label for the second image object;
process the one or more pixels to determine a classification for the first image object and the second image object, the classification being used in 3-dimensional transformation of the image for controlling operation of an Autonomous Vehicle (AV); and
associating a first classification label with the first image object and a second classification label with the second image object.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the one or more processors are further configured to execute the computer-readable instructions to:
process the first pixel region to determine a depth of the first image object based on LiDAR data.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the one or more processors are further configured to execute the computer-readable instructions to:
process the second pixel region to determine a depth of the second image object based on LiDAR data.
15. The non-transitory computer-readable storage medium of claim 12 , wherein the one or more processors are configure to execute the computer-readable instructions to place the bounding box around the first image object and the second image object is performed using a first machine-learning model.
16. The non-transitory computer-readable storage medium of claim 12 , wherein the one or more processors are configure to execute the computer-readable instructions to process the image area within the bounding box to identify the first image mask is performed using a second machine-learning model.