Multi-scale driving environment prediction with hierarchical spatial temporal attention
In accordance with one embodiment of the present disclosure, method includes obtaining multi-level environment data corresponding to a plurality of driving environment levels, encoding the multi-level environment data at each level, extracting features from the multi-level environment data at each encoded level, fusing the extracted features from each encoded level with a spatial-temporal attention framework to generate a fused information embedding, and decoding the fused information embedding to predict driving environment information at one or more driving environment levels.
1. A method comprising:
obtaining multi-level environment data corresponding to a plurality of driving environment levels;
encoding the multi-level environment data at each level;
extracting features from the multi-level environment data at each encoded level;
fusing the extracted features from each encoded level with a spatial-temporal attention framework to generate a fused information embedding; and
decoding the fused information embedding to predict driving environment information at one or more driving environment levels.
2. The method of claim 1 , wherein the multi-level environment data includes vehicle level data, local level data higher than the vehicle level data, and route level data higher than the local level data.
3. The method of claim 2 , wherein encoding the multi-level environment data at each level comprises one or more of:
encoding the vehicle level data to an occupancy grid representation;
encoding the local level data to a dynamic map representation; and
encoding the route level data to a graph representation.
4. The method of claim 2 , wherein extracting features from the multi-level environment data at each encoded level comprises one or more of:
extracting 3D environment features from an encoding of the vehicle level data;
extracting geometric features from an encoding of the local level data; and
extracting pattern features from an encoding of the route level data.
5. The method of claim 2 , wherein:
the vehicle level data is obtained from vehicle sensors of vehicles;
the local level data is obtained from sensors of roadside units; and
the route level data is obtained from a cloud server.
6. The method of claim 2 , wherein:
the vehicle level data is obtained with first latency;
the local level data is obtained with second latency longer than the first latency; and
the route level data is obtained with third latency longer than the second latency.
7. The method of claim 1 , wherein the spatial-temporal attention framework comprises:
concatenating the features from a pair of levels of the encoded multi-level environment data;
generating a plurality of input tensors based on the concatenated features; and
performing a transformer self-attention operation on the plurality of input tensors to generate an output information tensor.
8. The method of claim 1 , wherein decoding the fused information embedding to predict driving environment information at one or more driving environment levels comprises:
generating an optimal speed control;
generating an optimal lane selection; and
generating an optimal route selection.
9. The method of claim 1 , wherein extracting features from the multi-level environment data at each encoded level comprises extracting features from each encoded level at a plurality of scales of each encoded level.
10. A system comprising:
a controller programmed to perform operations comprising:
obtaining multi-level environment data corresponding to a plurality of driving environment levels;
encoding the multi-level environment data at each level;
extracting features from the multi-level environment data at each encoded level;
fusing the extracted features from each encoded level with a spatial-temporal attention framework to generate a fused information embedding; and
decoding the fused information embedding to predict driving environment information at one or more driving environment levels.
11. The system of claim 10 , wherein the multi-level environment data includes vehicle level data, local level data higher than the vehicle level data, and route level data higher than the local level data.
12. The system of claim 11 , wherein encoding the multi-level environment data at each level comprises one or more of:
encoding the vehicle level data to an occupancy grid representation;
encoding the local level data to a dynamic map representation; and
encoding the route level data to a graph representation.
13. The system of claim 11 , wherein extracting features from the multi-level environment data at each encoded level comprises one or more of:
extracting 3D environment features from an encoding of the vehicle level data;
extracting geometric features from an encoding of the local level data; and
extracting pattern features from an encoding of the route level data.
14. The system of claim 11 , wherein:
the vehicle level data is obtained from vehicle sensors of vehicles;
the local level data is obtained from sensors of roadside units; and
the route level data is obtained from a cloud server.
15. The system of claim 11 , wherein:
the vehicle level data is obtained with first latency;
the local level data is obtained with second latency longer than the first latency; and
the route level data is obtained with third latency longer than the second latency.
16. The system of claim 10 , wherein the spatial-temporal attention framework comprises:
concatenating the features from a pair of levels of the encoded multi-level environment data;
generating a plurality of input tensors based on the concatenated features; and
performing a transformer self-attention operation on the plurality of input tensors to generate an output information tensor.
17. The system of claim 10 , wherein decoding the fused information embedding to predict driving environment information at one or more driving environment levels comprises:
generating an optimal speed control;
generating an optimal lane selection; and
generating an optimal route selection.
18. The system of claim 10 , wherein extracting features from the multi-level environment data at each encoded level comprises extracting features from each encoded level at a plurality of scales of each encoded level.
19. A non-transitory medium having machine-readable instructions that cause a processor to perform operations comprising:
obtaining multi-level environment data corresponding to a plurality of driving environment levels;
encoding the multi-level environment data at each level;
extracting features from the multi-level environment data at each encoded level;
fusing the extracted features from each encoded level with a spatial-temporal attention framework to generate a fused information embedding; and
decoding the fused information embedding to predict driving environment information at one or more driving environment levels.
20. The non-transitory medium of claim 19 , wherein the multi-level environment data includes vehicle level data, local level data higher than the vehicle level data, and route level data higher than the local level data.