Vehicle trajectory prediction model with semantic map and LSTM
A system and method for predicting the near-term trajectory of a moving obstacle sensed by an autonomous driving vehicle (ADV) is disclosed. The method applies neural networks such as a LSTM model to learn dynamic features of the moving obstacle's motion based on its past trajectory up to its current position and a CNN model to learn the semantic map features of the driving environment in a portion of an image map. From the learned dynamic features of the moving obstacle and the learned semantic map features of the environment, the method applies a neural network to iteratively predict the moving obstacle's positions for successive time points of a prediction interval. To predict the moving obstacle's position at the next time point from the currently predicted position, the methods may update the learned dynamic features of the moving obstacle based on its past trajectory up to the currently predicted position.
1. A computer-implemented method to predict a trajectory of a target obstacle detected by an autonomous driving vehicles (ADV), the method comprising:
in response to an image of a driving environment around the target obstacle, processing the image by a first neural network (NN) to learn semantic map features of the image;
embedding features of the target obstacle into a low-dimensional vector;
processing the low-dimensional vector and most recent dynamic features of the target obstacle by a second NN to learn updated dynamic features of the target obstacle;
processing the updated dynamic features of the target obstacle and the semantic map features of the image by a third NN to generate a next predicted position of the target obstacle, the next predicted position including predicted features; and
repeating embedding the predicted features of the next predicted position, processing by the second NN and processing by the third NN until all predicted positions of a prediction interval are generated.
2. The method of claim 1 , wherein the features of the target obstacle in the image comprise one or more of a position, a heading, a speed, or a turning angle of the target obstacle.
3. The method of claim 1 , wherein the semantic map features of the image comprise one or more of vehicles, traffic elements, or road features in the image.
4. The method of claim 1 , wherein the most recent dynamic features of the target obstacle processed by the second NN to learn updated dynamic features of the target obstacle used to generate a first predicted position of the prediction interval comprises dynamic features of the target obstacle learned from a previous planning cycle of a plurality of planning cycles.
5. The method of claim 4 , wherein all predicted positions of the prediction interval are generated to correspond to a plurality of periodic time points during the prediction interval for each of the plurality of planning cycles.
6. The method of claim 1 , wherein the most recent dynamic features of the target obstacle processed by the second NN to learn updated dynamic features of the target obstacle used to generate a second or subsequent predicted position of the prediction interval comprises dynamic features of the target obstacle learned from a previous predicted position of the prediction interval.
7. The method of claim 1 , wherein the first NN comprises a convolutional NN (CNN) model, the second NN comprises a long short-term memory (LSTM) model, and the third NN comprises a multi-layer perceptron (MLP).
8. The method of claim 1 , wherein receiving an image containing the target obstacle comprises:
receiving an image map of one or more sensed obstacles detected by the ADV for a planning cycle;
selecting the target obstacle from the one or more sensed obstacles;
rotating the image map to position the target obstacle at a reference point in a reference heading of the rotated image map; and
cropping the rotated image map with respect to the target obstacle at the reference point in the reference heading to generate the image.
9. The method of claim 8 , further comprising:
selecting each of the one or more sensed obstacles as the target obstacle during the planning cycle.
10. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations to predict a trajectory of a target obstacle detected by an autonomous driving vehicles (ADV), the operations comprising:
in response to an image of a driving environment around the target obstacle, processing the image by a first neural network (NN) to learn semantic map features of the image;
embedding features of the target obstacle into a low-dimensional vector;
processing the low-dimensional vector and most recent dynamic features of the target obstacle by a second NN to learn updated dynamic features of the target obstacle;
processing the updated dynamic features of the target obstacle and the semantic map features of the image by a third NN to generate a next predicted position of the target obstacle, the next predicted position including predicted features; and
repeating embedding the predicted features of the next predicted position, processing by the second NN and processing by the third NN until all predicted positions of a prediction interval are generated.
11. The non-transitory machine-readable medium of claim 10 , wherein the features of the target obstacle in the image comprise one or more of a position, a heading, a speed, or a turning angle of the target obstacle.
12. The non-transitory machine-readable medium of claim 10 , wherein the semantic map features of the image comprise one or more of vehicles, traffic elements, or road features in the image.
13. The non-transitory machine-readable medium of claim 10 , wherein the most recent dynamic features of the target obstacle processed by the second NN to learn updated dynamic features of the target obstacle used to generate a first predicted position of the prediction interval comprises dynamic features of the target obstacle learned from a previous planning cycle of a plurality of planning cycles.
14. The non-transitory machine-readable medium of claim 10 , wherein the most recent dynamic features of the target obstacle processed by the second NN to learn updated dynamic features of the target obstacle used to generate a second or subsequent predicted position of the prediction interval comprises dynamic features of the target obstacle learned from a previous predicted position of the prediction interval.
15. The non-transitory machine-readable medium of claim 10 , wherein the first NN comprises a convolutional NN (CNN) model, the second NN comprises a long short-term memory (LSTM) model, and the third NN comprises a multi-layer perceptron (MLP).
16. The non-transitory machine-readable medium of claim 10 , wherein the operations further comprise:
receiving an image map of one or more sensed obstacles detected by the ADV for a planning cycle;
selecting the target obstacle from the one or more sensed obstacles;
rotating the image map to position the target obstacle at a reference point in a reference heading of the rotated image map; and
cropping the rotated image map with respect to the target obstacle at the reference point in the reference heading to generate the image.
17. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations of predicting a trajectory of a target obstacle detected by an autonomous driving vehicles (ADV), the operations comprising:
in response to an image of a driving environment around the target obstacle, processing the image by a first neural network (NN) to learn semantic map features of the image,
embedding features of the target obstacle into a low-dimensional vector;
processing the low-dimensional vector and most recent dynamic features of the target obstacle by a second NN to learn updated dynamic features of the target obstacle,
processing the updated dynamic features of the target obstacle and the semantic map features of the image by a third NN to generate a next predicted position of the target obstacle, the next predicted position including predicted features, and
repeating embedding the predicted features of the next predicted position, processing by the second NN and processing by the third NN until all predicted positions of a prediction interval are generated.
18. The data processing system of claim 17 , wherein the most recent dynamic features of the target obstacle processed by the second NN to learn updated dynamic features of the target obstacle used to generate a first predicted position of the prediction interval comprises dynamic features of the target obstacle learned from a previous planning cycle of a plurality of planning cycles.
19. The data processing system of claim 17 , wherein the first NN comprises a convolutional NN (CNN) model, the second NN comprises a long short-term memory (LSTM) model, and the third NN comprises a multi-layer perceptron (MLP).
20. The data processing system of claim 17 , wherein the operations further comprise:
receiving an image map of one or more sensed obstacles detected by the ADV for a planning cycle;
selecting the target obstacle from the one or more sensed obstacles;
rotating the image map to position the target obstacle at a reference point in a reference heading of a rotated image map; and
cropping the rotated image map with respect to the target obstacle at the reference point in the reference heading to generate the image.