Systems, methods, and media for semantic segmentation of a point cloud frame
A computer-implemented method, a computing system, and a non-transitory machine-readable medium for semantic segmentation of a point cloud frame are provided. Point cloud frames including a target point cloud frame are received. For each sequence of a sliding set of sequences of point cloud frames, the sequence including the target point cloud frame, each point cloud frame in the sequence of point cloud frames is semantically segmented to apply semantic labels to points. A most prevalent semantic label is determined for each point in the target point cloud frame across the sliding set of sequences of point cloud frames.
1. A computer-implemented method for semantic segmentation of a point cloud frame, comprising:
receiving point cloud frames including a target point cloud frame;
for each sequence of a sliding set of sequences of point cloud frames, the sequence including the target point cloud frame, semantically segmenting each point cloud frame in the sequence of point cloud frames to apply semantic labels to points; and
determining a most prevalent semantic label for each point in the target point cloud frame across the sliding set of sequences of point cloud frames.
2. The computer-implemented method of claim 1 , further comprising:
identifying a rare semantic label for at least one of the points in the target point cloud frame across the sliding set of sequences of point cloud frames; and
using the rare semantic label in place of the most prevalent semantic label for the at least one point in the target point cloud frame.
3. The computer-implemented method of claim 1 , further comprising:
performing a single-scan semantic segmentation of the target point cloud frame; and
using ensemble learning to combine the single-scan semantic segmentation with the most prevalent labels for each point in the target point cloud frame across the sliding set of sequences of point cloud frames.
4. The computer-implemented method of claim 2 , further comprising:
performing a single-scan semantic segmentation of the target point cloud frame; and
using ensemble learning to combine the single-scan semantic segmentation with the rare semantic labels and the most prevalent labels for each point in the target point cloud frame across the sliding set of sequences of point cloud frames.
5. The computer-implemented method of claim 1 , wherein the semantically segmenting includes:
voxelizing each of the point cloud frames in the sequence of point cloud frames;
generating a 4D tensor from the voxelized point cloud frames; and
processing the 4D tensor to identify features in the point cloud frames.
6. The computer-implemented method of claim 5 , wherein the processing includes:
for each point in the point cloud frames, finding k nearest neighbors; and
determining a semantic label for the point at least partially based on the features of the k nearest neighbors.
7. The computer-implemented method of claim 6 , wherein the determining includes:
calculating a weight for each of the k nearest neighbors at least partially based on a Euclidian distance between the point and the nearest neighbor.
8. The computer-implemented method of claim 7 , wherein, during the calculating of the Euclidian distance, a unit of distance is used to compensate for a unit of time between point cloud frames.
9. The computer-implemented method of claim 8 , wherein, during the determining, the weights are used to determine a weighted sum of the features of the nearest neighbors.
10. The computer-implemented method of claim 9 , wherein the weighted sum of the features of the nearest neighbors is used in combination with the features of the point to semantically segment the point.
11. A computing system for semantic segmentation of a point cloud frame, the computing system comprising:
a processor;
a memory storing machine-executable instructions that, when executed by the processor, cause the processor to:
receive point cloud frames including a target point cloud frame;
for each sequence of a sliding set of sequences of point cloud frames, the sequence including the target point cloud frame, semantically segment each point cloud frame in the sequence of point cloud frames to apply semantic labels to points; and
determine a most prevalent semantic label for each point in the target point cloud frame across the sliding set of sequences of point cloud frames.
12. The computing system of claim 11 , wherein the machine-executable instructions, when executed by the processor, cause the processor to:
identify a rare semantic label for at least one of the points in the target point cloud frame across the sliding set of sequences of point cloud frames; and
use the rare semantic label in place of the most prevalent semantic label for the at least one point in the target point cloud frame.
13. The computing system of claim 11 , wherein the machine-executable instructions, when executed by the processor, cause the processor to:
perform a single-scan semantic segmentation of the target point cloud frame; and
use ensemble learning to combine the single-scan semantic segmentation with the most prevalent labels for each point in the target point cloud frame across the sliding set of sequences of point cloud frames.
14. The computing system of claim 12 , wherein the machine-executable instructions, when executed by the processor, cause the processor to:
perform a single-scan semantic segmentation of the target point cloud frame; and
use ensemble learning to combine the single-scan semantic segmentation with the rare semantic labels and the most prevalent labels for each point in the target point cloud frame across the sliding set of sequences of point cloud frames.
15. The computing system of claim 11 , wherein the machine-executable instructions, when executed by the processor, cause the processor to, during semantic segmentation:
voxelize each of the point cloud frames in the sequence of point cloud frames;
generate a 4D tensor from the voxelized point cloud frames; and
process the 4D tensor to identify features in the point cloud frames.
16. The computing system of claim 15 , wherein the machine-executable instructions, when executed by the processor, cause the processor to, during the processing:
for each point in the point cloud frames, find k nearest neighbors; and
determine a semantic label for the point at least partially based on the features of the k nearest neighbors.
17. The computing system of claim 16 , wherein the machine-executable instructions, when executed by the processor, cause the processor to, during the determining:
calculate a weight for each of the k nearest neighbors at least partially based on a Euclidian distance between the point and the nearest neighbor.
18. The computing system of claim 17 , wherein the machine-executable instructions, when executed by the processor, cause the processor to, during the calculation of the Euclidian distance, use a unit of distance to compensate for a unit of time between point cloud frames.
19. The computing system of claim 18 , wherein the machine-executable instructions, when executed by the processor, cause the processor to, during the determining, use the weights to determine a weighted sum of the features of the nearest neighbors.
20. The computing system of claim 19 , wherein the weighted sum of the features of the nearest neighbors is used in combination with the features of the point to semantically segment the point.