Training a codebook for trajectory determination
Techniques for training a codebook usable by a machine learned model to predict an object trajectory or scene data are described herein. For example, the techniques may include generating tokens representing discrete object behavior into a machine learned model that outputs a sequence of tokens that is usable by another machine learned model to generate the object trajectory (e.g., position data, velocity data, acceleration data, etc.) or the scene data associated with the environment. The object trajectory can be sent to a vehicle computing device for consideration during vehicle planning, which may include simulation.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform actions comprising:
receiving, by a training component, state data representing a previous state of an object in an environment;
receiving, by the training component, feature vectors representing a vehicle and the object in the environment;
training a codebook as a trained codebook, the training comprising:
assigning, based at least in part on the state data, a first token to represent the previous state of the object;
assigning a second token to represent a characteristic of the vehicle;
assigning a third token to represent a feature of the environment; and
mapping the feature vectors to respective tokens; and
outputting the trained codebook for use by a machine learned model configured to access tokens from the trained codebook and to arrange the first token, the second token, and the third token to represent a potential interaction between the vehicle and the object.
2. The system of claim 1 , wherein the machine learned model is a transformer model, and the actions further comprising:
causing the transformer model to arrange the tokens based at least in part on a number of tokens and an order of the tokens.
3. The system of claim 1 , wherein:
the feature vectors represent continuous variables,
the tokens represent discrete latent variables, and
mapping the feature vectors to respective tokens comprises mapping the continuous variables associated with the feature vectors to the discrete latent variables associated with the tokens.
4. The system of claim 1 , the actions further comprising:
receiving, by the training component, environmental data representing the feature of the environment,
wherein training the codebook further comprises assigning the second token to represent the feature of the environment, and
the codebook is usable by the machine learned model to cause generation of a scene with the feature in the environment.
5. The system of claim 1 , wherein the vehicle is an autonomous vehicle.
6. A method comprising:
receiving, by a training component, state data representing a previous state of an object or a vehicle in an environment;
receiving, by the training component, feature vectors representing the vehicle and the object in the environment;
training a codebook to output a trained codebook, the training comprising:
assigning, based at least in part on the state data, a first token to represent the previous state of the object;
assigning a second token to represent a characteristic of the vehicle;
assigning a third token to represent a feature of the environment; and
mapping the feature vectors to respective tokens; and
outputting the trained codebook for use by a machine learned model configured to access tokens from the trained codebook and to arrange the first token, the second token, and the third token to represent a potential interaction between the vehicle and the object.
7. The method of claim 6 , further comprising:
determining a number of tokens to include in the codebook; and
determining an order of the tokens.
8. The method of claim 6 , further comprising:
causing the machine learned model to arrange the tokens based at least in part on a number of tokens and an order of the tokens.
9. The method of claim 6 , wherein:
the feature vectors represent continuous variables,
the tokens represent discrete latent variables, and
mapping the feature vectors to respective tokens comprises mapping the continuous variables associated with the feature vectors to the discrete latent variables associated with the tokens.
10. The method of claim 6 , further comprising:
receiving, by the training component, environmental data representing the feature of the environment,
wherein training the codebook further comprises assigning the second token to represent the feature of the environment, and
the codebook is usable by the machine learned model to cause generation of a scene with the feature in the environment.
11. The method of claim 6 , wherein the state data representing the previous state of the object or the vehicle comprising one or more of: position data, orientation data, heading data, velocity data, speed data, acceleration data, yaw rate data, or turning rate data.
12. The method of claim 6 , further comprising:
inputting image data into an encoder; and
outputting, by the encoder, the feature vectors,
wherein receiving the feature vectors by the training component comprises receiving the feature vectors from the encoder.
13. The method of claim 6 , wherein:
the feature vectors represent the characteristic of the vehicle and a characteristic of the object in the environment.
14. The method of claim 6 , wherein mapping the feature vectors to respective tokens comprises:
identifying an action or a state of the vehicle associated with a first feature vector of the feature vectors;
comparing the characteristic of the vehicle associated with the first feature vector to a characteristic associated with one or more of the tokens.
15. The method of claim 6 , wherein the vehicle is an autonomous vehicle.
16. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform actions comprising:
receiving, by a training component, state data representing a previous state of an object or a vehicle in an environment;
receiving, by the training component, feature vectors representing the vehicle and the object in the environment;
training a codebook to output a trained codebook, the training comprising:
assigning, based at least in part on the state data, a first token to represent the previous state of the object;
assigning a second token to represent a characteristic of the vehicle;
assigning a third token to represent a feature of the environment; and
mapping the feature vectors to respective tokens; and
outputting the trained codebook for use by a machine learned model configured to access tokens from the trained codebook and to arrange the first token, the second token, and the third token to represent a potential interaction between the vehicle and the object.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the machine learned model is a first machine learned model, and the actions further comprising:
causing the machine learned model to arrange the tokens based at least in part on a number of tokens and an order of the tokens.
18. The one or more non-transitory computer-readable media of claim 16 , wherein:
the feature vectors represent continuous variables,
the tokens represent discrete latent variables, and
mapping the feature vectors to respective tokens comprises mapping the continuous variables associated with the feature vectors to the discrete latent variables associated with the tokens.
19. The one or more non-transitory computer-readable media of claim 16 , the actions further comprising:
receiving, by the training component, environmental data representing the feature of the environment,
wherein training the codebook further comprises assigning the second token to represent the feature of the environment, and
the codebook is usable by the machine learned model to cause generation of a scene with the feature in the environment.
20. The one or more non-transitory computer-readable media of claim 16 , wherein the vehicle is an autonomous vehicle.