Point cloud filtering with semantic segmentation
Techniques are disclosed for filtering point cloud data associated with particulate matter (e.g., gas, exhaust, fog, etc.) which do not impact driving from data used to plan a trajectory and/or a route of a robotic platform. The filtering may be based on determining that a set of points associated with a point cloud represents a navigable space for the robotic platform. The point cloud data may be filtered based on a determination that one or more safety conditions are satisfied. Semantic segmentation may be performed on an image to determine pixel classification probability distributions associated with pixels of the image. Data associated with the set of points may be projected onto the image to identify corresponding pixels. Confidence scores associated with pixel classification probability distributions for the identified pixels may be queried. A classification probability distribution for the set of points may be determined based at least in part on the queried confidence scores.
1. An autonomous vehicle comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
receiving, from a LIDAR device on the autonomous vehicle, LIDAR data associated with an environment proximate the autonomous vehicle, the LIDAR data comprising a set of LIDAR points;
receiving, from a camera on the autonomous vehicle, image data associated with an image of a portion of the environment;
determining that the set of LIDAR points represents a non-navigable space;
projecting, as a LIDAR projection onto the image, a voxel associated with a point of the set of LIDAR points;
determining, based at least in part on a label associated with the image, a classification for a pixel associated with the voxel; and
determining, based at least in part on the classification, that the set of LIDAR points represents a navigable space for the autonomous vehicle.
2. The autonomous vehicle of claim 1 , wherein the instructions further cause the one or more processors to perform operations comprising:
projecting, as additional LIDAR projections onto the image, additional voxels associated with the set of LIDAR points;
determining additional classifications for pixels;
aggregating scores associated with the classification and additional classifications to produce an aggregation of scores; and
determining, based at least in part on the aggregation of scores, a classification for the set of LIDAR points.
3. The autonomous vehicle of claim 2 , wherein the determining that the set of LIDAR points represents a navigable space for the autonomous vehicle further comprises:
determining, based at least in part on the classification for the set of LIDAR points, to assign the set of LIDAR points to a class associated with a navigable space.
4. The autonomous vehicle of claim 1 , wherein the instructions further cause the one or more processors to perform operations comprising:
determining that one or more safety conditions are satisfied;
discarding, from data used to plan at least one of a trajectory or a route of the autonomous vehicle, LIDAR data associated with the set of LIDAR points, wherein the discarding is based at least in part on the determining that the set of LIDAR points represents a navigable space for the autonomous vehicle; and
controlling, based at least in part on the data used to plan the at least one of the trajectory or the route of the autonomous vehicle, movement of the autonomous vehicle.
5. The autonomous vehicle of claim 1 , wherein the determining the classification for the pixel associated with the voxel comprises:
determining, based at least in part on a semantic segmentation of the image, a classification probability distribution for the pixel associated with the voxel.
6. A computer-implemented method comprising:
receiving first data from a first sensor, the first data associated with a set of points of a three-dimensional (3D) point cloud of an environment;
receiving second data from a second sensor, the second data associated with an image associated with a portion of the environment;
determining that a portion of the set of points represents a non-navigable space;
associating the portion of the set of points with a portion of the second data;
determining a classification associated with the portion of the set of points; and
determining, based at least in part on the classification, that the portion of the set of points represents a navigable space for a vehicle.
7. The computer-implemented method of claim 6 , wherein the determining the classification associated with the portion of the set of points comprises:
projecting, onto the image, data associated with the portion of the set of points; and
determining, based at least in part on a label associated with the image, the classification associated with the portion of the set of points.
8. The computer-implemented method of claim 7 , wherein the determining the classification associated with the portion of the set of points comprises:
determining, based at least in part on a semantic segmentation of the image, a classification probability distribution associated with the portion of the set of points.
9. The computer-implemented method of claim 7 , further comprising:
projecting, onto the image, additional data associated with the portion of the set of points;
determining additional classifications for the portion of the set of points; and
aggregating scores associated with the classification and the additional classifications;
wherein the determining that the portion of the set of points represents a navigable space for the vehicle is based at least in part on the aggregating.
10. The computer-implemented method of claim 6 , wherein:
the vehicle comprises an autonomous vehicle; and
the computer-implemented method further comprises:
assigning the set of points to a class of object associated with a navigable space; and
discarding, from data used to plan at least one of a trajectory or a route of the autonomous vehicle, at least a portion of the first data associated with the set of points, wherein the discarding is based at least in part on the determining that the portion of the set of points represents a navigable space for the vehicle.
11. The computer-implemented method of claim 10 , further comprising:
controlling, based at least in part on the data used to plan the at least one of the trajectory or the route of the autonomous vehicle, movement of the autonomous vehicle.
12. The computer-implemented method of claim 10 , further comprising:
determining that one or more safety conditions are satisfied;
wherein the discarding the at least a portion of the first data associated with the set of points is further based at least in part on the determining that the one or more safety conditions are satisfied.
13. The computer-implemented method of claim 12 , wherein the one or more safety conditions comprise at least one of:
a first condition that a distance, between the set of points and the vehicle, satisfies a threshold distance;
a second condition that a speed of the vehicle satisfies a threshold speed;
a third condition that an amount of the set of points that projects onto the image satisfies a threshold amount; or
a fourth condition that a confidence level associated with the set of points belonging to a class associated with a navigable surface satisfies a threshold confidence level.
14. The computer-implemented method of claim 6 , wherein:
the set of points is a first set of points; and
the computer-implemented method further comprises:
receiving third data associated with a second set of points;
determining that the second set of points represents an obstacle to the vehicle; and
controlling, based at least in part on the third data, movement of the vehicle.
15. The computer-implemented method of claim 6 , wherein:
the first data is associated with a first time stamp;
the second data is associated with a second time stamp;
the computer-implemented method further comprises:
determining that an amount of time, between the first time stamp and the second time stamp, satisfies a threshold amount of time; and
the determining the classification associated with the portion of the set of points is based at least in part on the determining that the amount of time satisfies the threshold amount of time.
16. The computer-implemented method of claim 6 , further comprising:
inputting, into a machine learning algorithm, the first data associated with the set of points;
receiving, as output from the machine learning algorithm, a first score associated with a first probability that the set of points belongs to a first class of object;
determining, based at least in part on a semantic segmentation of the image, a second score associated with a second probability that the set of points belongs to a second class of object that is different from the first class of object;
determining that a difference between the first score and the second score satisfies a threshold difference; and
assigning, based at least in part on the determining that the difference between the first score and the second score satisfies the threshold difference, the set of points to the second class of object.
17. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
obtaining point cloud data comprising a set of points associated with measurements of an environment, the environment associated with a vehicle;
obtaining image data associated with an image comprising at least a portion of the environment associated with the set of points;
determining that a portion of the set of points represents a non-navigable space;
associating the portion of the set of points with a portion of the image data;
determining a classification associated with the portion of the set of points; and
determining, based at least in part on the classification, that the portion of the set of points represents a navigable space for the vehicle.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the determining the classification associated with the portion of the set of points comprises:
projecting, onto the image, data associated with the portion of the set of points; and
determining, based at least in part on a semantic segmentation of the image, a classification probability distribution associated with the portion of the set of points.
19. The one or more non-transitory computer-readable media of claim 17 , wherein the instructions further cause the one or more processors to perform operations comprising:
discarding, from data used to plan at least one of a trajectory or a route of the vehicle, at least a portion of the point cloud data associated with the set of points, wherein the discarding is based at least in part on the determining that the portion of the set of points represents a navigable space for the vehicle; and
controlling, based at least in part on the data used to plan the at least one of the trajectory or the route of the vehicle, movement of the vehicle.
20. The one or more non-transitory computer-readable media of claim 19 , wherein the instructions further cause the one or more processors to perform operations comprising:
determining that one or more safety conditions are satisfied;
wherein the discarding the at least a portion of the point cloud data associated with the set of points is further based at least in part on the determining that the one or more safety conditions are satisfied.