Semantic labeling of point clouds using images
Systems and methods for semantic labeling of point clouds using images. Some implementations may include obtaining a point cloud that is based on lidar data reflecting one or more objects in a space; obtaining an image that includes a view of at least one of the one or more objects in the space; determining a projection of points from the point cloud onto the image; generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image; inputting the augmented image to a two dimensional convolutional neural network to obtain a semantic labeled image wherein elements of the semantic labeled image include respective predictions; and mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud.
1. A system, comprising:
a data processing apparatus; and
a data storage device storing instructions executable by the data processing apparatus that upon execution by the data processing apparatus cause the data processing apparatus to perform operations comprising:
obtaining a point cloud in three spatial dimensions;
obtaining an image in two spatial dimensions;
determining a projection of points from the point cloud onto the image;
generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image;
inputting the augmented image to a neural network to obtain a semantic labeled image, wherein elements of the semantic labeled image include respective predictions; and
mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud.
2. The system of claim 1 , wherein the image is a first image and the semantic labeled image is a first semantic labeled image, and wherein the operations comprise:
obtaining a second image in two spatial dimensions;
determining a second semantic labeled image based on the second image augmented with data from the point cloud;
mapping predictions of the second semantic labeled image to respective points of the point cloud; and
accumulating predictions from the first semantic labeled image and from the second semantic labeled image for at least one point of the semantic labeled point cloud.
3. The system of claim 1 , wherein the operations comprise:
searching a set of images associated with different respective camera locations to identify a subset of images that includes at least two images with views of each point in the point cloud; and
wherein the image is obtained from the subset of images.
4. The system of claim 1 , wherein the operations comprise:
obtaining a training point cloud that includes points labeled with ground truth labels;
obtaining a training image, in two spatial dimensions, that includes a view of at least one object that is reflected in the training point cloud;
determining a projection of points from the training point cloud onto the training image;
generating, using the projection, an augmented training image that includes one or more channels of data from the training point cloud and one or more channels of data from the training image; and
training the neural network using the augmented training image and corresponding ground truth labels for projected points from the training point cloud.
5. The system of claim 1 , wherein the point cloud is determined using a bundle adjustment process based on lidar scans captured at a plurality of locations and times, and wherein the operations comprise:
assigning indications of moving likelihood to respective points of the point cloud based on how frequently the respective points are detected in lidar scans captured at different times;
applying a fully connected conditional random field to the indications of moving likelihood for points in the point cloud to obtain moving labels for respective points of the point cloud, wherein the moving labels are binary indications of whether or not a respective point of the point cloud corresponds to a moving object; and
wherein the moving labels are included in the augmented image as one of the one or more channels of data from the point cloud.
6. The system of claim 1 , wherein the operations comprise:
determining a graph based on the semantic labeled point cloud, wherein nodes of the graph are points from the semantic labeled point cloud and edges of the graph connect nodes with respective points that satisfy a pairwise criteria;
identifying one or more connected components of the graph; and
determining clusters of points from the semantic labeled point cloud by performing a hierarchical segmentation of each of the one or more connected components of the graph.
7. The system of claim 6 , wherein the operations comprise:
inputting predictions based on predictions for points of one of the clusters to a three dimensional convolutional neural network to obtain a prediction for the cluster; and
assigning the prediction for the cluster to all points of the cluster in the semantic labeled point cloud.
8. A method comprising:
obtaining a point cloud in three spatial dimensions;
obtaining an image in two spatial dimensions;
determining a projection of points from the point cloud onto the image;
generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image;
inputting the augmented image to a neural network to obtain a semantic labeled image, wherein elements of the semantic labeled image include respective predictions; and
mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud.
9. The method of claim 8 , wherein the image is a first image and the semantic labeled image is a first semantic labeled image, and further comprising:
obtaining a second image in two spatial dimensions;
determining a second semantic labeled image based on the second image augmented with data from the point cloud;
mapping predictions of the second semantic labeled image to respective points of the point cloud; and
accumulating predictions from the first semantic labeled image and from the second semantic labeled image for at least one point of the semantic labeled point cloud.
10. The method of claim 8 , comprising:
searching a set of images associated with different respective camera locations to identify a subset of images that includes at least two images with views of each point in the point cloud; and
wherein the image is obtained from the subset of images.
11. The method of claim 8 , comprising:
obtaining a training point cloud that includes points labeled with ground truth labels;
obtaining a training image, in two spatial dimensions, that includes a view of at least one object that is reflected in the training point cloud;
determining a projection of points from the training point cloud onto the training image;
generating, using the projection, an augmented training image that includes one or more channels of data from the training point cloud and one or more channels of data from the training image; and
training the neural network using the augmented training image and corresponding ground truth labels for projected points from the training point cloud.
12. The method of claim 8 , wherein the point cloud is determined using a bundle adjustment process based on lidar scans captured at a plurality of locations and times, and comprising:
assigning indications of moving likelihood to respective points of the point cloud based on how frequently the respective points are detected in lidar scans captured at different times;
apply a fully connected conditional random field to the indications of moving likelihood for points in the point cloud to obtain moving labels for respective points of the point cloud, wherein the moving labels are binary indications of whether or not a respective point of the point cloud corresponds to a moving object; and
wherein the moving labels are included in the augmented image as one of the one or more channels of data from the point cloud.
13. The method of claim 8 , wherein the one or more channels of data from the point cloud that are included in the augmented image include at least one channel from amongst the set of depth, normal, height, lidar intensity, lidar variance, and spin.
14. The method of claim 8 , comprising:
determining a graph based on the semantic labeled point cloud, wherein nodes of the graph are points from the semantic labeled point cloud and edges of the graph connect nodes with respective points that satisfy a pairwise criteria;
identifying one or more connected components of the graph; and
determining clusters of points from the semantic labeled point cloud by performing a hierarchical segmentation of each of the one or more connected components of the graph.
15. The method of claim 14 , comprising:
inputting predictions based on predictions for points of one of the clusters to a three dimensional convolutional neural network to obtain a prediction for the cluster; and
assigning the prediction for the cluster to all points of the cluster in the semantic labeled point cloud.
16. The method of claim 15 , wherein the predictions input to the three dimensional convolutional neural network are associated with respective voxels that collectively form a block centered at a center of the one of the clusters, and wherein the predictions input to the three dimensional convolutional neural network are determined as an average of predictions for points located within a respective voxel.
17. The method of claim 8 , comprising:
applying a fully connected conditional random field to the predictions of the semantic labeled point cloud to refine the predictions.
18. The method of claim 8 , comprising:
scaling a channel of the augmented image to have dynamic range matching another channel of the augmented image.
19. The method of claim 8 , comprising:
training the neural network using a loss function that includes a term that is a function of depth.
20. A non-transitory computer-readable storage medium including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:
obtaining a point cloud in three spatial dimensions;
obtaining an image in two spatial dimensions;
determining a projection of points from the point cloud onto the image;
generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image;
inputting the augmented image to a neural network to obtain a semantic labeled image, wherein elements of the semantic labeled image include respective predictions; and
mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud.