Object detection and tracking, including parallelized lidar semantic segmentation
Generating a semantic segmentation of point cloud data may include detecting an object based at least in part one or more images and using the image-based object detection in combination with the point cloud data to determine the semantic segmentation of the point cloud data. The semantic segmentation of point cloud data may be conducted in parallel with executing an object tracking component. Parallelizing these functions reduces the amount of time it takes to use sensors to detect and track objects and the point cloud semantic segmentation may provide further granularity for planning and ML model training.
1 . A method comprising:
receiving sensor data, wherein the sensor data comprises image data and lidar data comprising three-dimensional lidar points;
determining, based on the image data, an object detection indicating:
a portion of the image data that is associated with an object in an environment being traversed by a vehicle, and
a first object classification associated with the portion;
determining, based on projecting a first lidar point of the three-dimensional lidar points to a two-dimensional space associated with the image data, a projected lidar point in the two-dimensional space;
determining, based at least in part on whether the projected lidar point falls within the portion of the image data, a probability that the first lidar point is associated with the first object classification;
determining, based at least in part on the probability meeting or exceeding a probability threshold, that the first lidar point is included in a subset of the three-dimensional lidar points that is associated with the first object classification,
wherein the subset of the three-dimensional lidar points is determined based on projecting the subset to the two-dimensional space associated with the image data;
determining a three-dimensional region of interest (ROI) that includes a threshold portion of the subset of the three-dimensional lidar points; and
controlling the vehicle based at least in part on the three-dimensional ROI.
2 . The method of claim 1 , wherein determining the object detection comprises:
determining, by a machine-learned model and based on the image data as input, the portion of the image data and the first object classification.
3 . A system comprising:
one or more processors; and
a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving sensor data, wherein the sensor data comprises image data and point cloud data;
receiving an object detection associated with an object in an environment being traversed by a vehicle, the object detection indicating:
a portion of the image data associated with the object, and
a first object classification associated with the object;
determining, based on projecting a first point of the point cloud data, from a three-dimensional space associated with the point cloud data into a two-dimensional space associated with the image data, a probability that the first point is associated with the first object classification;
determining, based at least in part on the probability meeting or exceeding a probability threshold, that the first point is included in a subset of points of the point cloud data that is associated with the first object classification,
wherein the subset of the points is determined based on projecting the subset to the two-dimensional space associated with the image data;
determining a three-dimensional region of interest (ROI) that includes a threshold portion of the subset of the points; and
controlling the vehicle based at least in part on the three-dimensional ROI.
4 . The system of claim 3 , wherein the point cloud data comprises at least one of lidar data, radar data, or time-of-flight data.
5 . The system of claim 3 , wherein the object detection is a first object detection and the image data is captured by a first sensor type,
the operations further comprising:
receiving a second object detection based at least in part on a second sensor type, the first object detection and the second object detection identifying the object;
receiving a previous track associated with the object, the previous track identifying at least one of an estimated previous position of the object, a previous region of interest, or a previous velocity of the object;
inputting the first object detection, the second object detection, and at least part of the previous track into a machine-learned model; and
receiving, from the machine-learned model, a data structure comprising a region of interest and a second object classification associated with the object; and
determining, based at least in part on the data structure and a current position of the object, a current track associated with the object.
6 . The system of claim 5 , wherein the machine-learned model is trained to output a grid associated with an area in the environment having a minimum distance of more than thirty meters from the vehicle to any point on an edge of the area.
7 . The system of claim 5 , wherein:
determining the probability is accomplished by a first component of the vehicle;
determining the current track is accomplished by a second component of the vehicle; and
the first component and the second component operate in parallel.
8 . The system of claim 3 , determining the probability comprises:
determining a projected point in the two-dimensional space corresponding to the first point; and
determining that the projected point falls within the portion of the image data associated with the first object classification.
9 . A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving sensor data, wherein the sensor data comprises image data and point cloud data representing an environment associated with a vehicle;
determining, an object detection indicating a classification of an object in the environment and a portion of the image data associated with the object, wherein the classification includes an identification of an object type;
determining, based on projecting a first point of the point cloud data into a two-dimensional space associated with the image data, a probability that the first point is associated with the classification;
determining, based at least in part on the probability meeting or exceeding a probability threshold, that the first point is included in a subset of points of the point cloud data that is associated with the classification,
wherein the subset of the points is determined based on projecting the subset to the two-dimensional space associated with the image data;
determining a three-dimensional region of interest (ROI) that includes a threshold portion of the subset of the points; and
controlling the vehicle based at least in part on the three-dimensional ROI.
10 . The non-transitory computer-readable medium of claim 9 , wherein:
the classification of the object is determined by a machine-learned model based at least in part on the image data as input, and
determining the probability is based on determining that projection of the first point falls within the portion of the image data.
11 . The non-transitory computer-readable medium of claim 9 , wherein the object detection is a first object detection and the image data is captured by a first sensor type,
the operations further comprising:
receiving a second object detection associated with a second sensor type, the first object detection and the second object detection identifying the object;
receiving a previous track associated with the object, the track identifying at least one of an estimated previous position of the object, a previous region of interest, or a previous velocity of the object;
inputting the first object detection, the second object detection, and at least part of the previous track into a second machine-learned model; and
receiving, from the second machine-learned model, a data structure comprising a region of interest and object classification associated with the object; and
determining, based at least in part on the data structure and a current position of the object, a current track associated with the object.
12 . The non-transitory computer-readable medium of claim 11 , wherein:
determining the probability is accomplished by a first component of the vehicle;
determining the track is accomplished by a second component of the vehicle; and
the first component and the second component operate in parallel.
13 . The method of claim 1 , wherein the probability is determined by a first component of a perception component associated with the vehicle, the method further comprising:
determining, based at least in part on the object detection and by a second component of the perception component, a track associated with the object,
wherein determining the probability and determining the track are separately executed in parallel by the first component and the second component.
14 . The method of claim 13 , wherein:
the track is a first track associated with a first time, and
the probability is determined based at least in part on a second track associated with the object, the second track determined at a second time prior to the first time.
15 . The system of claim 3 , wherein the probability is a first probability determined at a first time, the operations further comprising:
accessing a second probability, determined at a second time prior to the first time, the second probability indicative of a position being associated with the object,
wherein the first probability is determined at least in part on the second probability.
16 . The system of claim 15 , wherein the second probability is determined based at least in part on a track associated with the object, the track including an estimated position of the object or a velocity of the object.
17 . The system of claim 3 , the operations further comprising:
determining a track associated with the object, wherein the track associates the object detection with one or more previous object detections,
wherein the controlling the vehicle is further based at least in part on the track.
18 . The system of claim 3 , wherein determining the probability is based on determining whether the projected first point is located at a position, in the two-dimensional space, that is within an area of the two-dimensional space corresponding to the portion of the image data associated with the object.
19 . The non-transitory computer-readable medium of claim 9 , the operations further comprising:
determining, in parallel with determining the probability, a track associated with the object, wherein the track associates the object detection with one or more previous object detections, wherein the controlling the vehicle is further based at least in part on the track.
20 . The non-transitory computer-readable medium of claim 9 , wherein determining the probability is based on determining whether the projected first point is located at a position, in the two-dimensional space, that is within an area of the two-dimensional space corresponding to the portion of the image data associated with the object.