End-to-end processing in automated driving systems
The described aspects and implementations enable efficient object detection and tracking. In one implementation, disclosed is a method and a system to perform the method, the system including the sensing system configured to obtain sensing data characterizing an environment of the vehicle. The system further includes a data processing system operatively coupled to the sensing system and configured to process the sensing data using a first (second) set of neural network (NN) layers to obtain a first (second) set of features for a first (second) region of the environment, the first (second) set of features is associated with a first (second) spatial resolution. The data processing system is further to process the two sets of features using a second set of NN layers to detect a location of object(s) in the environment of the vehicle and a state of motion of the object(s).
1 . A method comprising:
obtaining, by a processing device, an input data characterizing an environment of a vehicle, wherein the input data comprises at least lidar sensing data and camera sensing data; and
inputting, by the processing device, the lidar sensing data into a lidar neural network (NN) that generates:
a first set of lidar feature vectors respectively associated with a first plurality of pixels of a first pixel size, and
a second set of lidar feature vectors respectively associated with a second plurality of pixels of a second pixel size, the second pixel size being less than the first pixel size;
inputting, by the processing device, the camera sensing data into a camera NN that generates:
a first set of camera feature vectors respectively associated with the first plurality of pixels of the first pixel size, and
a second set of camera feature vectors respectively associated with the second plurality of pixels of the second pixel size:
aggregating at least the first set of camera feature vectors and the first set of lidar feature vectors to obtain a first set of aggregated feature vectors associated with the first pixel size;
aggregating at least the second set of camera feature vectors and the second set of lidar feature vectors to obtain a second set of aggregated feature vectors associated with the second pixel size;
processing, by the processing device, the first set of aggregated feature vectors and the second set of aggregated feature vectors to generate:
a first set of object detections associated with the first pixel size, and
a second set of object detections associated with the second pixel size;
identifying one or more objects in the environment of the vehicle based at least on aggregation of the first set of object detections and the second set of object detections; and
controlling a driving path of the vehicle in view of the one or more identified objects.
2 . The method of claim 1 , wherein the first plurality of pixels and the second plurality of pixels are associated with a bird's eye view (BEV) coordinate system, and wherein the first set of camera feature vectors and the second set of camera feature vectors are generated using a lifting transform to the BEV from a perspective view associated with the camera sensing data.
3 . The method of claim 1 , wherein the input data further comprises radar sensing data, the method further comprising:
inputting, by the processing device, the radar sensing data into a radar NN that generates:
a first set of radar feature vectors respectively associated with the first plurality of pixels of the first pixel size, and
a second set of radar feature vectors respectively associated with the second plurality of pixels of the second pixel size;
wherein the first set of aggregated feature vectors is obtained by further aggregating the first set of radar feature vectors, and wherein the second set of aggregated feature vectors is obtained by further aggregating the second set of radar feature vectors.
4 . The method of claim 1 , wherein the first plurality of pixels of
the first pixel size depicts a first region of the environment, and
wherein the second plurality of pixels of the second pixel size depicts a second region of the environment, wherein the second region is smaller than the first region.
5 . The method of claim 1 , wherein the input data further comprises roadgraph data that maps a drivable portion of the environment of the vehicle, the method further comprising:
inputting, by the processing device, the roadgraph data into a roadgraph NN that generates:
a first set of roadgraph feature vectors respectively associated with the first plurality of pixels of the first pixel size, and
a second set of roadgraph feature vectors respectively associated with the second plurality of pixels of the second pixel size;
wherein the first set of aggregated feature vectors is obtained by further aggregating the first set of roadgraph feature vectors, and wherein the second set of aggregated feature vectors is obtained by further aggregating the second set of roadgraph feature vectors, and wherein processing the first set of aggregated feature vectors and the second set of aggregated feature vectors is further to generate a current state of the drivable portion of the environment of the vehicle.
6 . The method of claim 5 , wherein the current state of the drivable portion of the environment of the vehicle comprises a status of one or more traffic lights in the environment of the vehicle.
7 . The method of claim 1 , wherein the first set of aggregated feature vectors and the second set of aggregated feature vectors are processed using a common backbone and a plurality of classifier heads receiving inputs generated by the common backbone, wherein the plurality of classifier heads comprises one or more of:
a segmentation head,
an occupancy head,
a traffic flow head,
an object occlusion head, or
a roadgraph head.
8 . The method of claim 1 , wherein processing the first set of aggregated feature vectors and the second set of aggregated feature vectors is further to generate:
a state of motion of a first object of the one or more objects.
9 . The method of claim 8 , wherein the state of motion of the first object is identified for a plurality of times, the method further comprising:
predicting, using at least the state of motion of the first object for the plurality of times, a trajectory of the first object.
10 . The method of claim 1 , wherein the first set of aggregated feature vectors comprises a first subset of aggregated feature vectors associated with a first time and a second subset of aggregated feature vectors associated with a second time, and wherein processing the first set of aggregated feature vectors and the second set of aggregated feature vectors comprises:
performing spatial warping of the first subset of aggregated feature vectors from the first time to the second time; and
aggregating the first subset of aggregated feature vectors, spatially warped, with the second subset of aggregated feature vectors.
11 . The method of claim 1 , wherein the aggregation of the first set of object detections and the second set of object detections comprises using non-maximum suppression (NMS).
12 . A system comprising:
a sensing system of a vehicle, the sensing system configured to:
obtain an input data characterizing an environment of the vehicle, wherein the input data comprises at least lidar sensing data and camera sensing data; and
a perception system of the vehicle, the perception system configured to:
input the lidar sensing data into a lidar neural network (NN) that generates:
a first set of lidar feature vectors respectively associated with a first plurality of pixels of a first pixel size, and
a second set of lidar feature vectors respectively associated with a second plurality of pixels of a second pixel size, the second pixel size being less than the first pixel size;
input the camera sensing data into a camera NN that generates:
a first set of camera feature vectors respectively associated with the first plurality of pixels of the first pixel size, and
a second set of camera feature vectors respectively associated with the second plurality of pixels of the second pixel size;
aggregate at least the first set of camera feature vectors and the first set of lidar feature vectors to obtain a first set of aggregated feature vectors associated with the first pixel size;
aggregate at least the second set of camera feature vectors and the second set of lidar feature vectors to obtain a second set of aggregated feature vectors associated with the second pixel size;
process the first set of aggregated feature vectors and the second set of aggregated feature vectors to generate:
a first set of object detections associated with the first pixel size, and
a second set of object detections associated with the second pixel size;
identify one or more objects in the environment of the vehicle based at least on aggregation of the first set of object detections and the second set of object detections; and
control a driving path of the vehicle in view of the one or more identified objects.
13 . The system of claim 12 , wherein the first plurality of pixels and the second plurality of pixels are associated with a bird's eye view (BEV) coordinate system, and wherein the first set of camera feature vectors and the second set of camera feature vectors are generated using a lifting transform to the BEV from a perspective view associated with the camera sensing data.
14 . The system of claim 13 , wherein the input data further comprises radar sensing data, and wherein the perception system is further configured to:
input the radar sensing data into a radar NN that generates:
a first set of radar feature vectors respectively associated with the first plurality of pixels of the first pixel size, and
a second set of radar feature vectors respectively associated with the second plurality of pixels of the second pixel size;
wherein the first set of aggregated feature vectors is obtained by further aggregating the first set of radar feature vectors, and wherein the second set of aggregated feature vectors is obtained by further aggregating the second set of radar feature vectors.
15 . The system of claim 12 , wherein the input data further comprises roadgraph data that maps a drivable portion of the environment of the vehicle, wherein the perception system is further configured to:
input the roadgraph data into a roadgraph NN that generates:
a first set of roadgraph feature vectors respectively associated with the first plurality of pixels of the first pixel size, and
a second set of roadgraph feature vectors respectively associated with the second plurality of pixels of the second pixel size;
wherein the first set of aggregated feature vectors is obtained by further aggregating the first set of roadgraph feature vectors, and wherein the second set of aggregated feature vectors is obtained by further aggregating the second set of roadgraph feature vectors, and wherein to process the first set of aggregated feature vectors and the second set of aggregated feature vectors, the perception system is further configured to generate a current state of the drivable portion of the environment of the vehicle.
16 . The system of claim 15 , wherein the current state of the drivable portion of the environment of the vehicle comprises a status of one or more traffic lights in the environment of the vehicle.
17 . The system of claim 12 , wherein the first set of aggregated feature vectors and the second set of aggregated feature vectors are processed using a common backbone and a plurality of classifier heads receiving inputs generated by the common backbone, wherein the plurality of classifier heads comprises one or more of:
a segmentation head,
an occupancy head,
a traffic flow head,
an object occlusion head, or
a roadgraph head.
18 . The system of claim 12 , wherein to process the first set of aggregated feature vectors and the second set of aggregated feature vectors, the perception system is further to identify a state of motion of a first object of the one or more objects for a plurality of times, and wherein the perception system is further to:
predict, using at least the state of motion of the first object for the plurality of times, a trajectory of the first object.
19 . The system of claim 12 , wherein the aggregation of the first set of object detections and the second set of object detections comprises using non-maximum suppression (NMS).
20 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to:
obtain an input data characterizing an environment of a vehicle, wherein the input data comprises at least lidar sensing data and camera sensing data; and
input the lidar sensing data into a lidar neural network (NN) that generates:
a first set of lidar feature vectors respectively associated with a first plurality of pixels of a first pixel size, and
a second set of lidar feature vectors respectively associated with a second plurality of pixels of a second pixel size, the second pixel size being less than the first pixel size;
input the camera sensing data into a camera NN that generates:
a first set of camera feature vectors respectively associated with the first plurality of pixels of the first pixel size, and
a second set of camera feature vectors respectively associated with the second plurality of pixels of the second pixel size;
aggregate at least the first set of camera feature vectors and the first set of lidar feature vectors to obtain a first set of aggregated feature vectors associated with the first pixel size;
aggregate at least the second set of camera feature vectors and the second set of lidar feature vectors to obtain a second set of aggregated feature vectors associated with the second pixel size:
process the first set of aggregated feature vectors and the second set of aggregated feature vectors to generate:
a first set of object detections associated with the first pixel size, and
a second set of object detections associated with the second pixel size;
identify one or more objects in the environment of the vehicle based at least on aggregation of the first set of object detections and the second set of object detections; and
control a driving path of the vehicle in view of the one or more identified objects.