Semantic segmentation of three-dimensional data
Systems and methods for performing semantic segmentation of three-dimensional data are provided. In one example embodiment, a computing system can be configured to obtain sensor data including three-dimensional data associated with an environment. The three-dimensional data can include a plurality of points and can be associated with one or more times. The computing system can be configured to determine data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data. The computing system can be configured to determine a classification for each point of the plurality of points within the three-dimensional data based at least in part on the two-dimensional voxel representation associated with the environment and a machine-learned semantic segmentation model. The computing system can be configured to initiate one or more actions based at least in part on the per-point classifications.
1. A computer-implemented method of semantic segmentation, the method comprising:
obtaining sensor data comprising three-dimensional data associated with an environment;
determining data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data and a voxel grid, wherein the two-dimensional voxel representation is associated with one or more voxels;
determining a voxel classification for respective voxels of the one or more voxels associated with the two-dimensional voxel representation, based at least in part on the two-dimensional voxel representation; and
determining a classification for respective points of a plurality of points within the three-dimensional data based at least in part on the voxel classification for the respective voxels.
2. The computer-implemented method of claim 1 , wherein determining a voxel classification for a respective voxel comprises:
determining that a voxel of the one or more voxels is associated with an object in the environment; and
determining a classification of the object based on the voxel classification.
3. The computer-implemented method of claim 2 , wherein the classification is indicative of (i) a static object or (ii) a dynamic object.
4. The computer-implemented method of claim 1 , further comprising:
determining that the sensor data is associated with a plurality of times; and
determining the classification for the respective points across the plurality of times.
5. The computer-implemented method of claim 4 , wherein the classification for the respective points across the plurality of times is associated with motion of one or more objects in the environment.
6. The computer-implemented method of claim 1 , further comprising determining a classification of one or more portions of the environment based on the voxel classification for the respective voxels.
7. The computer-implemented method of claim 1 , further comprising:
determining a probability distribution across a plurality of classes for the respective voxels; and
predicting the voxel classification for the respective voxels based on the probability distribution.
8. The computer-implemented method of claim 1 , wherein the voxel grid comprises a plurality of voxel cells.
9. A vehicle computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors to perform operations, the operations comprising:
obtaining sensor data comprising three-dimensional data associated with an environment;
determining data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data and a voxel grid, wherein the two-dimensional voxel representation is associated with one or more voxels;
determining a voxel classification for respective voxels of the one or more voxels associated with the two-dimensional voxel representation, based at least in part on the two-dimensional voxel representation; and
determining a classification for respective points of a plurality of points within the three-dimensional data based at least in part on the voxel classification for the respective voxels.
10. The vehicle computing system of claim 9 , wherein determining a voxel classification for a respective voxel comprises:
determining that a voxel of the one or more voxels is associated with an object in the environment; and
determining a classification of the object based on the voxel classification.
11. The vehicle computing system of claim 10 , wherein the classification is indicative of (i) a static object or (ii) a dynamic object.
12. The vehicle computing system of claim 9 , further comprising:
determining that the sensor data is associated with a plurality of times; and
determining the classification for the respective points across the plurality of times.
13. The vehicle computing system of claim 12 , wherein the classification for the respective points across the plurality of times is associated with motion of one or more objects in the environment.
14. The vehicle computing system of claim 9 , further comprising determining a classification of one or more portions of the environment based on the voxel classification for the respective voxels.
15. The vehicle computing system of claim 9 , further comprising:
determining a probability distribution across a plurality of classes for the respective voxels; and
predicting the voxel classification for the respective voxels based on the probability distribution.
16. The vehicle computing system of claim 9 , wherein the voxel grid comprises a plurality of voxel cells.
17. One or more non-transitory computer-readable media storing instructions executable by one or more processors to cause the one or more processors to perform operations, the operations comprising:
obtaining sensor data comprising three-dimensional data associated with an environment;
determining data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data and a voxel grid, wherein the two-dimensional voxel representation is associated with one or more voxels;
determining a voxel classification for respective voxels of the one or more voxels associated with the two-dimensional voxel representation, based at least in part on the two-dimensional voxel representation; and
determining a classification for respective points of a plurality of points within the three-dimensional data based at least in part on the voxel classification for the respective voxels.
18. The one or more non-transitory computer-readable media of claim 17 , wherein determining a voxel classification for a respective voxel comprises:
determining that a voxel of the one or more voxels is associated with an object in the environment; and
determining a classification of the object based on the voxel classification.
19. The one or more non-transitory computer-readable media of claim 18 , wherein the classification is indicative of (i) a static object or (ii) a dynamic object.
20. The one or more non-transitory computer-readable media of claim 17 , further comprising:
determining that the sensor data is associated with a plurality of times; and
determining the classification for the respective points across the plurality of times.