Training a variable autoencoder using a diffusion model
Techniques for training a variable autoencoder to output data associated with one or more objects in an environment are described herein. For example, the techniques can include training an encoder and a decoder of the variable autoencoder to improve predictions over time. The variable autoencoder can be trained to output object occupancy information (e.g., a bounding box, heatmap, feature vector) and/or object attribute information (e.g., an object state, an object type, etc.). A vehicle computing device can use an output from a trained autoencoder during vehicle planning, which may include simulation.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
inputting, as first input data into an encoder of a variable autoencoder, map data representing an environment and occupancy data associated with an object in the environment;
receiving, from the encoder, first output data representing a compressed representation of the first input data;
receiving, from a diffusion model, discrete latent variable data associated with the object, the diffusion model configured to implement a diffusion process to add or remove noise from condition data received as input;
inputting, as second input data into a decoder of the variable autoencoder, the first output data from the encoder and the discrete latent variable data from the diffusion model;
receiving, from the decoder, second output data representing an occupancy representation for the object in the environment and object state data associated with the object; and
training the encoder or the decoder based at least in part on the second output data.
2 . The system of claim 1 , wherein the discrete latent variable data associated with the object indicates an acceleration action, a braking action, or a steering action of the object.
3 . The system of claim 1 , wherein:
the map data is associated with a simulated environment,
the compressed representation of the first input data represents a latent embedding of the first input data, and
the object state data indicates an acceleration, a velocity, an orientation, or a position of the object.
4 . The system of claim 1 , the operations further comprising:
performing a simulation to verify a response by a vehicle relative to the object; and
training, based at least in part on the response, the decoder to output bounding box data for the object.
5 . The system of claim 1 , the operations further comprising:
transmitting the second output data to a vehicle computing device to control a vehicle in the environment.
6 . One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
inputting, into an encoder, map data representing an environment and occupancy data associated with an object in the environment;
receiving, from the encoder, first output data representing a compressed representation of the map data and the occupancy data;
receiving, from a diffusion model, discrete latent variable data associated with the object, the diffusion model configured to implement a diffusion process to add or remove noise from condition data received as input;
inputting, into a decoder, the first output data from the encoder and the discrete latent variable data from the diffusion model;
receiving, from the decoder, second output data representing an occupancy representation for the object in the environment and object state data associated with the object; and
training the encoder or the decoder based at least in part on the second output data.
7 . The one or more non-transitory computer-readable media of claim 6 , wherein the second output data comprises a trajectory, a velocity, an acceleration, or an orientation associated with the object.
8 . The one or more non-transitory computer-readable media of claim 6 , wherein:
the map data is associated with a simulated environment,
the compressed representation represents a latent embedding of data input to the encoder, and
the object state data indicates at least one of: a trajectory, an acceleration, a velocity, an orientation, width or length of the object, or a position of the object.
9 . The one or more non-transitory computer-readable media of claim 6 , the operations further comprising:
performing a simulation to verify a response by a vehicle relative to the object; and
training, based at least in part on the response, the decoder to output bounding box data for the object.
10 . The one or more non-transitory computer-readable media of claim 6 , the operations further comprising:
transmitting the second output data to a vehicle computing device to control a vehicle in the environment.
11 . The one or more non-transitory computer-readable media of claim 6 , wherein the encoder and the decoder are components of a variable autoencoder.
12 . The one or more non-transitory computer-readable media of claim 6 , wherein training the encoder or the decoder comprises:
comparing, as a comparison, the first output data or the second output data to ground truth; and
training the encoder or the decoder based at least in part on the comparison.
13 . The one or more non-transitory computer-readable media of claim 6 , wherein the occupancy representation comprises a bounding box to represent the object in the environment.
14 . The one or more non-transitory computer-readable media of claim 6 , wherein the occupancy representation comprises a feature vector to represent the object in the environment.
15 . The one or more non-transitory computer-readable media of claim 6 , wherein:
the object is a first object,
the second output data comprises:
a first bounding box associated with the first object,
a second bounding box associated with a second object, and
an orientation or a position of the first object.
16 . The one or more non-transitory computer-readable media of claim 6 , wherein the occupancy representation comprises a heatmap to represent the object in the environment.
17 . A method comprising:
inputting, into an encoder, map data representing an environment and occupancy data associated with an object in the environment;
receiving, from the encoder, first output data representing a compressed representation of the map data and the occupancy data;
receiving, from a diffusion model, discrete latent variable data associated with the object, the diffusion model configured to implement a diffusion process to add or remove noise from condition data received as input;
inputting, into a decoder, the first output data from the encoder and the discrete latent variable data from the diffusion model;
receiving, from the decoder, second output data representing an occupancy representation for the object in the environment and object state data associated with the object; and
training the encoder or the decoder based at least in part on the second output data.
18 . The method of claim 17 , wherein the second output data comprises a trajectory, a velocity, or an acceleration associated with the object.
19 . The method of claim 17 , wherein:
the map data is associated with a simulated environment,
the compressed representation represents a latent embedding of data input to the encoder, and
the object state data indicates an acceleration, a velocity, an orientation, or a position of the object.
20 . The method of claim 17 , further comprising:
performing a simulation to verify a response by a vehicle relative to the object; and
training, based at least in part on the response, the decoder to output bounding box data for the object.