Systems and methods for generating motion forecast data for a plurality of actors with respect to an autonomous vehicle
A computing system can input first relative location embedding data into an interaction transformer model and receive, as an output of the interaction transformer model, motion forecast data for actors relative to a vehicle. The computing system can input the motion forecast data into a prediction model to receive respective trajectories for the actors for a current time step and respective projected trajectories for the actors for a subsequent time step. The computing system can generate second relative location embedding data based on the respective projected trajectories from the second time step. The computing system can produce second motion forecast data using the interaction transformer model based on the second relative location embedding. The computing system can determine second respective trajectories for the actors using the prediction model based on the second forecast data.
1. A computing system, comprising:
a machine-learned interaction transformer model configured to receive a relative location embedding that describes relative respective locations of a plurality of actors with respect to an autonomous vehicle, wherein the relative location embedding encodes the respective relative locations of the plurality of actors as a multi-channel positional embedding, and in response to receipt of the relative location embedding, generate motion forecast data describing movement of the plurality of actors;
a machine-learned prediction model configured to receive the motion forecast data, and in response to receipt of the motion forecast data, generate respective trajectories for the plurality of actors for a current time step and respective projected trajectories for a subsequent time step;
a memory that stores a set of instructions; and
one or more processors which use the set of instructions to:
for a first iteration corresponding with a first time step:
generate first motion forecast data for a plurality of actors at the first time step using the machine-learned interaction transformer model based on a first relative location embedding;
input the first motion forecast data into the machine-learned prediction model; and
receive, as an output of the machine-learned prediction model, first respective trajectories of the plurality of actors for the first time step and respective projected trajectories for a second time step that is after the first time step; and
for a second iteration corresponding with the second time step:
generate a second relative location embedding for the second time step based on the respective projected trajectories for the second time step from the first iteration;
analyze the second relative location embedding using the machine-learned interaction transformer model to produce, as an output of the machine-learned interaction transformer model, second motion forecast data for the plurality of actors at the second time step; and
determine second respective trajectories of the plurality of actors for the second time step using the machine-learned prediction model based on the second motion forecast data.
2. The computing system of claim 1 , wherein:
the machine-learned interaction transformer model is configured to generate the motion forecast data by receiving the relative location embedding that describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle and by further receiving an input sequence describing object detection data, and in response to receipt of the relative location embedding and the input sequence, generate the motion forecast data with respect to the plurality of actors;
the first motion forecast data is based on the first relative location embedding and a first input sequence corresponding with the first time step; and
the second motion forecast data is produced by using the machine-learned interaction transformer model to analyze the second relative location embedding and a second input sequence corresponding with the second time step.
3. The computing system of claim 2 , wherein the second input sequence is based at least in part on the first motion forecast data.
4. The computing system of claim 1 , wherein the instructions further comprise:
generate respective trajectory sequences for the plurality of actors, the respective trajectory sequences comprising the first respective trajectories of the plurality of actors for the first time step and the second respective trajectories of the plurality of actors for the second time step.
5. The computing system of claim 2 , wherein the second input sequence further describe one or more features for the plurality of actors respectively at the first time step or the second time step, the one or more features comprising at least one of:
a derivative of the relative respective location of a respective actor of the plurality of actors relative to the autonomous vehicle;
a size of the respective actor;
an orientation of the respective actor relative to the autonomous vehicle; or
a center location of the respective actor.
6. The computing system of claim 1 , wherein:
the first relative location embedding describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle at the first time step; and
the second relative location embedding describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle at the second time step.
7. The computing system of claim 1 , wherein the machine-learned interaction transformer model comprises:
a machine-learned interaction model configured to receive the relative location embedding that describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle, and in response to receipt of the relative location embedding, generate an attention embedding with respect to the plurality of actors;
a machine-learned recurrent model configured to receive the attention embedding, and in response to receipt of the attention embedding, generate the motion forecast data with respect to the plurality of actors.
8. The computing system of claim 1 , wherein at least one of the machine-learned interaction transformer model or the machine-learned prediction model comprises one or more neural networks.
9. The computing system of claim 1 , wherein the machine-learned prediction model comprises one or more multi-layer perceptrons.
10. A computer-implemented method, the method comprising:
for a first iteration corresponding with a first time step:
inputting, by a computing system comprising one or more computing devices, a first relative location embedding that describes relative respective locations of a plurality of actors with respect to an autonomous vehicle into a machine-learned interaction transformer model that is configured to receive a relative location embedding, wherein the relative location embedding encodes the respective relative locations of the plurality of actors as a multi-channel positional embedding, and in response to receipt of the relative location embedding, generate motion forecast data describing movement of the plurality of actors;
receiving, by the computing system and as an output of the machine-learned interaction transformer model, first motion forecast data for a first plurality of actors at the first time step;
inputting, by the computing system, the first motion forecast data into a machine-learned prediction model, the machine-learned prediction model configured to receive motion forecast data, and in response to receipt of the motion forecast data, generate respective trajectories for the plurality of actors for a current time step and respective projected trajectories for a subsequent time step; and
receiving, by the computing system and as an output of the machine-learned prediction model, first respective trajectories of the plurality of actors for the first time step and respective projected trajectories for a second time step that is after the first time step; and
for a second iteration corresponding with the second time step:
generating, by the computing system, a second relative location embedding for the second time step based on the respective projected trajectories for the second time step from the first iteration;
analyzing, by the computing system using the machine-learned interaction transformer model, the second relative location embedding to produce, as an output of the machine-learned interaction transformer model, second motion forecast data for the plurality of actors at the second time step; and
determining, by the computing system using the machine-learned prediction model, second respective trajectories of the plurality of actors for the second time step using the machine-learned prediction model based on the second motion forecast data.
11. The computer-implemented method of claim 10 , wherein:
the machine-learned interaction transformer model is configured to generate the motion forecast data by receiving the relative location embedding that describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle and by further receiving an input sequence describing object detection data, and in response to receipt of the relative location embedding and the input sequence, generate the motion forecast data with respect to the plurality of actors;
the first motion forecast data is based on the first relative location embedding and a first input sequence corresponding with the first time step; and
the second motion forecast data is produced by using, by the computing system, the machine-learned interaction transformer model to analyze the second relative location embedding and a second input sequence corresponding with the second time step.
12. The computer-implemented method of claim 11 , wherein the second input sequence is based at least in part on the first motion forecast data.
13. The computer-implemented method of claim 10 , further comprising generating, by the computing system, respective trajectory sequences for the plurality of actors, the respective trajectory sequences comprising the first respective trajectories of the plurality of actors for the first time step and the second respective trajectories of the plurality of actors for the second time step.
14. The computer-implemented method of claim 11 , wherein the second input sequence further describe one or more features for the plurality of actors respectively at the first time step or the second time step, the one or more features comprising at least one of:
a derivative of the relative respective location of a respective actor of the plurality of actors relative to the autonomous vehicle;
a size of the respective actor;
an orientation of the respective actor relative to the autonomous vehicle; or
a center location of the respective actor.
15. The computer-implemented method of claim 10 , wherein;
the first relative location embedding describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle at the first time step; and
the second relative location embedding describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle at the second time step.
16. The computer-implemented method of claim 10 , wherein the machine-learned interaction transformer model comprises:
a machine-learned interaction model configured to receive the relative location embedding that describes the relative respective locations of the plurality of actors with respect to the autonomous vehicle, and in response to receipt of the relative location embedding, generate an attention embedding with respect to the plurality of actors; and
a machine-learned recurrent model configured to receive the attention embedding, and in response to receipt of the attention embedding, generate the motion forecast data with respect to the plurality of actors.
17. The computer-implemented method of claim 10 , wherein the machine-learned prediction model comprises one or more multi-layer perceptrons.
18. A computer-implemented method for training one or more machine-learned models, the method comprising:
for a first iteration corresponding with a first time step:
inputting, by a computing system comprising one or more computing devices, a first relative location embedding that describes relative respective locations of a plurality of actors with respect to an autonomous vehicle into a machine-learned interaction transformer model that is configured to receive a relative location embedding, and in response to receipt of the relative location embedding, generate motion forecast data describing movement of the plurality of actors;
receiving, by the computing system and as an output of the machine-learned interaction transformer model, first motion forecast data for a first plurality of actors at the first time step;
inputting, by the computing system, the first motion forecast data into a machine-learned prediction model, the machine-learned prediction model configured to receive motion forecast data, and in response to receipt of the motion forecast data, generate respective trajectories for the plurality of actors for a current time step and respective projected trajectories for a subsequent time step; and
receiving, by the computing system and as an output of the machine-learned prediction model, first respective trajectories of the plurality of actors for the first time step and respective projected trajectories for a second time step that is after the first time step; and
for a second iteration corresponding with the second time step:
generating, by the computing system, a second relative location embedding for the second time step based on the respective projected trajectories for the second time step from the first iteration;
analyzing, by the computing system using the machine-learned interaction transformer model, the second relative location embedding to produce, as an output of the machine-learned interaction transformer model, second motion forecast data for the plurality of actors at the second time step;
determining, by the computing system using the machine-learned prediction model, second respective trajectories of the plurality of actors for the second time step using the machine-learned prediction model based on the second motion forecast data; and
adjusting, by the computing system, one or more parameters of the machine-learned interaction transformer model and the machine-learned prediction model based on the second respective trajectories of the plurality of actors.
19. The computer-implemented method of claim 18 , wherein adjusting, by the computing system, one or more parameters of the machine-learned interaction transformer model and the machine-learned prediction model based on the second respective trajectories of the plurality of actors comprises adjusting, by the computing system, the one or more parameters of the machine-learned interaction transformer model and the machine-learned prediction model based on a loss function that describes a difference between respective ground truth trajectories of the plurality of actors and the second respective trajectories of the plurality of actors for the second time step.
20. The computer-implemented method of claim 18 , wherein adjusting, by the computing system, one or more parameters of the machine-learned interaction transformer model and the machine-learned prediction model based on the second respective trajectories of the plurality of actors comprises training, in an end-to-end configuration, the machine-learned interaction transformer model and the machine-learned prediction model.