Utilizing multiple machine learning models with autonomous vehicles
The described aspects and implementations enable efficient utilization of multiple machine learning (ML) models with autonomous vehicles (AVs) to quickly and efficiently determine trajectory information of objects in a driving environment. During a time interval, a perception system of an AV generates, using a first ML model, an embedding based on data that characterizes an object's trajectory in the driving environment. At each sub-interval of multiple sub-intervals during the time interval, a second ML model uses the embedding to generate a probability distribution for the trajectory of the object, and a planning system of the AV generates an update to the AV's trajectory based on the probability distribution.
1 . A method, comprising:
generating, using a first machine learning (ML) model, a first embedding, wherein generating the first embedding is based on an initial trajectory of an object in a driving environment around an autonomous vehicle (AV);
at successive times during operation of the AV, generating, using a second ML model and the first embedding generated before the successive times, a corresponding probability distribution over a current trajectory of the object at a respective successive time, wherein the current trajectory of the object varies for each generation of the corresponding probability distribution; and
providing each corresponding probability distribution to a planning system of the AV to cause the planning system to repetitively generate an update to a trajectory of the AV.
2 . The method of claim 1 , further comprising autonomously modifying operations of the AV based on each update to the trajectory of the AV.
3 . The method of claim 1 , wherein each corresponding probability distribution comprises:
a plurality of estimated current trajectories of the object; and
for each of the plurality of estimated current trajectories, a corresponding probability.
4 . The method of claim 1 , wherein:
the first ML model comprises a first number of artificial neurons and the second ML model comprises a second number of artificial neurons; and
the first number is at least ten times larger than the second number.
5 . The method of claim 1 , wherein the first embedding is further based on at least one of:
a geometry of the driving environment around the AV; or
a traffic signal state in the driving environment around the AV.
6 . The method of claim 1 , wherein:
the first embedding is further based on an initial trajectory of the AV; and
the method further comprises generating, using the second ML model and based on the first embedding, a future trajectory of the AV.
7 . The method of claim 1 , wherein:
generating the first embedding occurs at a beginning of a first time interval that comprises a plurality of sub-intervals; and
each generation of the corresponding probability distribution occurs at a respective one of the plurality of sub-intervals.
8 . The method of claim 7 , wherein:
the plurality of sub-intervals comprises a first sub-interval and a second sub-interval that occurs after the first sub-interval; and
the generation of the corresponding probability distribution at the second sub-interval includes using the corresponding probability distribution generated at the first sub-interval as input to the second ML model.
9 . The method of claim 8 , further comprising, at the second time interval that occurs after the first time interval:
generating, using the first ML model, a second embedding, wherein generating the second embedding is based on the current trajectory of the object;
repetitively generating, using the second ML model and the second embedding, the corresponding probability distribution for the current trajectory of the object; and
providing each corresponding probability distribution to the planning system of the AV to repetitively generate further updates to the trajectory of the AV.
10 . The method of claim 7 , wherein each sub-interval of the plurality of sub-intervals is less than 300 milliseconds.
11 . A system, comprising:
a memory; and
one or more processing devices, coupled to the memory, to perform operations comprising:
generating, using a first machine learning (ML) model, a first embedding, wherein generating the first embedding is based on an initial trajectory of an object in a driving environment around an autonomous vehicle (AV);
at successive times during operation of the AV, generating, using a second ML model and the first embedding generated before the successive times, a corresponding probability distribution over a current trajectory of the object at a respective successive time, wherein the current trajectory of the object varies for each generation of the corresponding probability distribution; and
providing each corresponding probability distribution to a planning system of the AV to cause the planning system to repetitively generate an update to a trajectory of the AV.
12 . The system of claim 11 , wherein:
the one or more processing devices comprise a first processing device and a second processing device;
the first ML model executes on the first processing device; and
the second ML model executes on the second processing device.
13 . The system of claim 11 , wherein:
the system further comprises a sensing system of the AV, wherein the sensing system comprises one or more sensors configured to acquire a position of the object; and
generating the first embedding is further based on the position.
14 . The system of claim 11 , wherein:
the operations further comprise obtaining a first positional history of the object; and
generating the first embedding is further based on the first positional history.
15 . The system of claim 14 , wherein the first positional history of the object comprises a plurality of coordinates of the object and a velocity of the object.
16 . The system of claim 14 , wherein:
generating the first embedding and generating the corresponding probability distribution occur during a first time interval; and
the operations further comprise:
obtaining a second positional history of the object,
during a second time interval that occurs after the first time interval, generating, using the first ML model, a second embedding, wherein generating the second embedding is based on the current trajectory of the object and the second positional history of the object, and
during the second time interval, repetitively generating, using the second ML model and the second embedding, the corresponding probability distribution for the current trajectory of the object.
17 . A non-transitory computer storage media storing instructions that when executed by one or more processing devices cause the one or more processing devices to perform operations comprising:
during a first time interval, generating, using a first subnetwork of a first machine learning (ML) model, a first embedding, wherein generating the first embedding is based on a trajectory of an object in a driving environment around an autonomous vehicle (AV);
during a second time interval, generating, using a second subnetwork of the first ML model and based on the first embedding, a second embedding; and
at successive times during the second time interval, generating, using a second ML model and based on the second embedding, a corresponding probability distribution over the trajectory of the object at a respective successive time; and
providing the corresponding probability distribution to a planning system of the AV to cause the planning system to generate an update to a trajectory of the AV.
18 . The computer storage media of claim 17 , wherein generating the first embedding and generating the second embedding comprise executing the first subnetwork and the second subnetwork on a graphics processing unit (GPU).
19 . The computer storage media of claim 17 , wherein the operations further comprise:
obtaining one or more camera images of the driving environment around the AV; and
determining, based on the one or more camera images, the trajectory of the object.
20 . The computer storage media of claim 17 , wherein:
the first embedding comprises a first vector that includes a first number of elements;
the second embedding comprises a second vector that includes a second number of elements; and
the first number is different from the second number.