Three-dimensional point clouds based on images and depth data
Techniques are discussed herein for generating three-dimensional (3D) representations of an environment based on two-dimensional (2D) image data, and using the 3D representations to perform 3D object detection and other 3D analyses of the environment. 2D image data may be received, along with depth estimation data associated with the 2D image data. Using the 2D image data and associated depth data, an image-based object detector may generate 3D representations, including point clouds and/or 3D pixel grids, for the 2D image or particular regions of interest. In some examples, a 3D point cloud may be generated by projecting pixels from the 2D image into 3D space followed by a trained 3D convolutional neural network (CNN) performing object detection. Additionally or alternatively, a top-down view of a 3D pixel grid representation may be used to perform object detection using 2D convolutions.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising:
receiving two-dimensional image data captured by a camera in an environment, the two-dimensional image data including a first pixel and a second pixel;
inputting the two-dimensional image data to a two-dimensional object detection algorithm;
determining, based at least in part on the two-dimensional object detection algorithm, that the first pixel and the second pixel are associated with an object;
determining that a third pixel of the two-dimensional image data is outside of a first portion associated with the object;
determining, based at least in part on an output of a depth estimation algorithm, first estimated depth data associated with the first pixel and second estimated depth data associated with the second pixel, wherein the output of the depth estimation algorithm is generated based on an input of the two-dimensional image data, and wherein the first estimated depth data is different from the second estimated depth data;
quantizing the first estimated depth data into a first predetermined depth quanta, before projecting the first pixel; and
quantizing the second depth data into a second predetermined depth quanta, before projecting the second pixel;
projecting a subset of pixels within a fist two-dimensional bounding box, the subset of pixels comprising less than all of the pixels of the two-dimensional image data into a three-dimensional space, the projecting comprising:
projecting the first pixel as a first point into a three-dimensional space, based at least in part on the first estimated depth data, a pixel projection calculation for the first pixel, and the first pixel being associated with the object;
projecting the second pixel as a second point into the three-dimensional space, based at least in part on the second estimated depth data, the pixel projection calculation for the second pixel, and the second pixel being associated with the object; and
excluding the third pixel form a projecting operation, based at least in part on determining that the third pixel is outside of the first portion of the two-dimensional image data;
generating a point cloud associated with the two-dimensional image data, the point cloud including the first point and the second point;
voxelizing at least a portion of the point cloud into a plurality of grids with a plurality of pillars being formed on the plurality of grids, the plurality of pillars including the first point and the second point;
providing the plurality of grids with the plurality of pillars to a three-dimensional object detection algorithm;
determining the object in the environment, based at least in part on an output of the three-dimensional object detection algorithm;
back-projecting an associated three-dimensional bounding box generated based on the point cloud into a two-dimensional plane of the two-dimensional image data to obtain a back-projected two-dimensional bounding box; and
evaluating an accuracy of determining the object in the environment based at least in part on a result of comparison between the first two-dimensional bounding box and the back-projected two-dimensional bounding box.
2 . The system as recited in claim 1 , wherein the first portion includes the first pixel and the second pixel.
3 . The system as recited in claim 1 , the operations further comprising:
modifying the associated three-dimensional bounding box, based at least in part on comparing the first two-dimensional bounding box and the back-projected two-dimensional bounding box.
4 . The system as recited in claim 1 , wherein the first estimated depth data and the second estimated depth data is determined without associated captured depth data.
5 . The system of claim 1 , the operations further comprising determining an instance mask associated with the object.
6 . The system of claim 5 , wherein determining the instance mask associated with the object comprises determining the instance mask tailored to a profile of the object.
7 . A method comprising:
receiving two-dimensional image data captured by a camera, the two-dimensional image data comprising a set of pixels that includes a first pixel and a second pixel;
inputting the two-dimensional image data to a two-dimensional object detection algorithm;
determining, based at least in part on the two-dimensional object detection algorithm, that the first pixel and the second pixel are associated with an object;
determining that a third pixel of the two-dimensional image data is outside of a first portion associated with the object;
determining, based on inputting the two-dimensional image data to a depth estimation algorithm, first estimated depth data associated with the first pixel and second estimated depth data associated with the second pixel;
quantizing the first estimated depth data into a first predetermined depth quanta, before projecting the first pixel;
quantizing the second depth data into a second predetermined depth quanta, before projecting the second pixel;
projecting a subset of pixels within a fist two-dimensional bounding box, the subset of pixels comprising less than all of the pixels of the two-dimensional image data into a three-dimensional space, the projecting comprising:
projecting the first pixel as a first point into a three-dimensional space, based at least in part on the first estimated depth data, and the first pixel being associated with the object;
projecting the second pixel as a second point into the three-dimensional space, based at least in part on the second predetermined depth quanta, and the second pixel being associated the object; and
excluding the third pixel form a projecting operation, based at least in part on determining that the third pixel is outside of the first portion of the two-dimensional image data;
generating a point cloud associated with the two-dimensional image data, the point cloud including the first point and the second point;
voxelizing at least a portion of the point cloud into a plurality of grids with a plurality of pillars being formed on the plurality of grids, the plurality of pillars including the first point and the second point;
determining the object in an environment, based at least in part on an output of a three-dimensional object detection algorithm;
back-projecting an associated three-dimensional bounding box generated based on the point cloud into a two-dimensional plane of the two-dimensional image data to obtain a back-projected two-dimensional bounding box; and
evaluating an accuracy of determining the object in the environment based at least in part on a result of comparison between the first two-dimensional bounding box and the back-projected two-dimensional bounding box.
8 . The method of claim 7 , further comprising:
providing the plurality of grids with the plurality of pillars to the three-dimensional object detection algorithm.
9 . The method of claim 7 , wherein first portion includes the first pixel and the second pixel.
10 . The method of claim 7 , further comprising:
modifying the associated three-dimensional bounding box, based at least in part on comparing the first two-dimensional bounding box and the back-projected two-dimensional bounding box.
11 . The method of claim 7 , wherein the first estimated depth data and the second estimated depth data is determined without associated captured depth data.
12 . The method of claim 7 , further comprising determining an instance mask associated with the object.
13 . The method of claim 12 , wherein determining the instance mask associated with the object comprises determining the instance mask tailored to a profile of the object.
14 . One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving two-dimensional image data captured by a camera, the two-dimensional image data including a first pixel and a second pixel;
inputting the two-dimensional image data to a two-dimensional object detection algorithm;
determining, based at least in part on the two-dimensional object detection algorithm, that the first pixel and the second pixel are associated with an object;
determining that a third pixel of the two-dimensional image data is outside of a first portion associated with the object;
determining, based at least in part on an output of a depth estimation algorithm, first estimated depth data associated with the first pixel and second estimated depth data associated with the second pixel, wherein the output of the depth estimation algorithm is generated based on input of the two-dimensional image data;
quantizing the first estimated depth data into a first predetermined depth quanta, before projecting the first pixel;
quantizing the second estimated depth data into a second predetermined depth quanta, before projecting the second pixel;
projecting a subset of pixels within a fist two-dimensional bounding box, the subset of pixels comprising less than all of the pixels of the two-dimensional image data into a three-dimensional space, the projecting comprising:
projecting the first pixel as a first point into a three-dimensional space, based at least in part on the first estimated depth data, and the first pixel being associated with the object;
projecting the second pixel as a second point into the three-dimensional space, based at least in part on the second predetermined depth quanta, and the second pixel being associated the object;
excluding the third pixel form a projecting operation, based at least in part on determining that the third pixel is outside of the first portion of the two-dimensional image data;
generating a point cloud associated with the two-dimensional image data, the point cloud including the first point and the second point; and
voxelizing at least a portion of the point cloud into a plurality of grids with a plurality of pillars being formed on the plurality of grids, the plurality of pillars including the first point and the second point;
determining the object in an environment, based at least in part on an output of a three-dimensional object detection algorithm;
back-projecting an associated three-dimensional bounding box generated based on the point cloud into a two-dimensional plane of the two-dimensional image data to obtain a back-projected two-dimensional bounding box; and
evaluating an accuracy of determining the object in the environment based at least in part on a result of comparison between the first two-dimensional bounding box and the back-projected two-dimensional bounding box.
15 . The one or more non-transitory computer-readable media of claim 14 , the operations further comprising:
providing the plurality of grids with the plurality of pillars to the three-dimensional object detection algorithm; and
determining an object type associated with the first pixel, based at least in part on the output of the three-dimensional object detection algorithm.
16 . The one or more non-transitory computer-readable media of claim 14 , wherein the first portion includes the first pixel and the second pixel.
17 . The one or more non-transitory computer-readable media of claim 14 , wherein the first estimated depth data and the second estimated depth data is determined without associated captured depth data.
18 . The one or more non-transitory computer-readable media of claim 14 , the operations further comprising determining an instance mask associated with the object.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein determining the instance mask associated with the object comprises determining the instance mask tailored to a profile of the object.