Method for predicting a trajectory of an agent in a vicinity of a self-driving vehicle based on ranking
A method of operating a self-driving vehicle is provided. The method includes predicting a trajectory of an agent in a vicinity of the self-driving vehicle. This is done by: receiving sensor data indicative of a current situation in the vicinity of the self-driving vehicle; generating a current situation feature vector based, at least in part, on the sensor data; searching a pre-built bank of library feature vectors to identify a result feature vector from the pre-built bank that is most relevant to the current situation feature vector, wherein each library feature vector in the pre-built bank is associated with an observed trajectory; and predicting the trajectory of the agent based, at least in part, on the result feature vector. The method also includes planning an action of the vehicle based, at least in part, on the predicted agent trajectory, and operating the vehicle according to the planned action.
1. A method of generating a bank of library feature vectors for use in predicting a trajectory of an agent in a vicinity of an in-use self-driving vehicle, the method comprising:
receiving, on a server, a plurality of observation samples, each observation sample collected by a sensor on a data collection self-driving vehicle, each observation sample including data on a trajectory of an observed agent in the vicinity of the data collection self-driving vehicle over a predetermined period of time;
generating a library of agent trajectories by:
clustering agent trajectories from at least a first portion of the plurality of observation samples into a plurality of clusters, each cluster in the plurality of clusters containing similar agent trajectories,
randomly selecting a cluster from the plurality of clusters, and
randomly selecting an agent trajectory, to be added to the library of agent trajectories, from the selected cluster;
generating, on the server, a library feature vector for each trajectory in the library of agent trajectories;
assembling, on the server, the bank of library feature vectors based, at least in part, on the library feature vectors, wherein each library feature vector in the bank of library feature vectors represents a possible future trajectory for the agent in the vicinity of the data collection self-driving vehicle; and
sending the bank of library feature vectors to a computing device of the in-use self-driving vehicle.
2. The method of claim 1 , wherein assembling, on the server, the bank of library feature vectors further comprises generating an index of library feature vectors in the bank of library feature vectors.
3. The method of claim 2 , wherein generating the index of library feature vectors comprises generating a proximity graph of the library feature vectors.
4. The method of claim 3 , wherein the proximity graph comprises a hierarchical navigable small world graph.
5. The method of claim 1 , wherein generating, on the server, the library feature vector for each trajectory in the library of agent trajectories comprises using a neural network to generate the library feature vector from each trajectory in the library of agent trajectories.
6. The method of claim 1 , wherein each observation sample further includes data on a scene surrounding the observed agent, and wherein the method further comprises:
using at least a second portion of the plurality of observation samples for training a first neural network to generate first feature vectors based on input of the scenes surrounding the observed agents; and
using at least the second portion of the plurality of observation samples for jointly training a second neural network to generate second feature vectors based on input of the trajectories of the observed agents; and
wherein the first feature vectors and second feature vectors will be close when the trajectory of an observed agent corresponds to the scene surrounding that observed agent.
7. The method of claim 1 , further comprising:
predicting, by the computing device of the in-use self-driving vehicle, a trajectory of an agent in a vicinity of the in-use self-driving vehicle by:
receiving sensor data indicative of a current situation in the vicinity of the in-use self-driving vehicle from a sensor associated with the in-use self-driving vehicle;
generating a current situation feature vector based, at least in part, on the sensor data;
searching the bank of library feature vectors to identify a result feature vector from the bank of library feature vectors that is most relevant to the current situation feature vector; and
predicting the trajectory of the agent based, at least in part, on the result feature vector;
planning, by the computing device of the in-use self-driving vehicle, an action of the in-use self-driving vehicle based, at least in part, on the predicted trajectory of the agent; and
operating the in-use self-driving vehicle according to the planned action.
8. The method of claim 7 , wherein generating the current situation feature vector comprises using a pre-trained neural network to generate the current situation feature vector.
9. The method of claim 8 , wherein using the pre-trained neural network comprises providing a feature map representing aspects of a birds-eye view of the current situation in the vicinity of the in-use self-driving vehicle, based, at least in part, on the sensor data as input to the pre-trained neural network.
10. The method of claim 9 , wherein the feature map is centered on the agent, and has an orientation based on an orientation of the agent.
11. The method of claim 7 , wherein searching the bank of library feature vectors comprises performing an approximate nearest neighbor search.
12. The method of claim 11 , wherein the bank of library feature vectors includes an index based on a hierarchical navigable small world graph, and wherein performing the approximate nearest neighbor search comprises traversing the hierarchical navigable small world graph.
13. The method of claim 7 , wherein the result feature vector comprises the library feature vector that is closest to the current situation feature vector.
14. The method of claim 7 , wherein searching the bank of library feature vectors comprises generating a ranked list of feature vectors from the bank of library feature vectors.
15. The method of claim 14 , wherein generating the ranked list of feature vectors comprises ranking the library feature vectors based, at least in part, on a measure of relevance of the library feature vectors to the current situation feature vector.
16. A server comprising:
a processor;
a memory coupled to the processor, the memory comprising programmed instructions that when executed by the processor cause the processor to:
access, on the server, a plurality of observation samples, each observation sample collected by a sensor on a data collection self-driving vehicle, each observation sample including data on a trajectory of an observed agent in a vicinity of the data collection self-driving vehicle over a predetermined period of time;
generate a library of agent trajectories by:
clustering agent trajectories from at least a first portion of the plurality of observation samples into a plurality of clusters, each cluster in the plurality of clusters containing similar agent trajectories,
randomly selecting a cluster from the plurality of clusters, and
randomly selecting an agent trajectory, to be added to the library of agent trajectories, from the selected cluster;
generate, on the server, a library feature vector for each trajectory in the library of agent trajectories;
assemble, on the server, a bank of library feature vectors based, at least in part, on the library feature vectors, wherein each library feature vector in the bank of library feature vectors represents a possible future trajectory for the agent in the vicinity of the data collection self-driving vehicle; and
provide the bank of library feature vectors to a computing device of a self-driving vehicle.
17. A system comprising the server of claim 16 and the self-driving vehicle, wherein the self-driving vehicle comprises:
at least one sensor;
a computing device including a processor and a memory, the memory storing programmed instructions that when executed by the processor cause the computing device to:
predict a trajectory of an agent in a vicinity of the self-driving vehicle by:
receiving sensor data indicative of a current situation in the vicinity of the self-driving vehicle from the at least one sensor;
generating a current situation feature vector based, at least in part, on the sensor data;
searching the bank of library feature vectors to identify a result feature vector from the bank of library feature vectors that is most relevant to the current situation feature vector; and
predicting the trajectory of the agent based, at least in part, on the result feature vector;
plan an action of the self-driving vehicle based, at least in part, on the predicted trajectory of the agent; and
operate the self-driving vehicle according to the planned action.
18. The system of claim 17 , wherein the self-driving vehicle comprising a self-driving car.