Generating object representations using a variable autoencoder
Techniques for generating a representation for an object in an environment of an autonomous vehicle are described herein. For example, the techniques may include a decoder of a variable autoencoder receiving latent variable data from a diffusion model and determining an object representation such as a bounding box or a heatmap for one or more objects proximate the autonomous vehicle. The bounding box can include orientation data indicating an orientation for each of the one or more bounding boxes. The object representation(s) can be sent to a vehicle computing device for consideration during vehicle planning, which may include simulation.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
inputting, as a first input into a decoder of a variable autoencoder, map data representing an environment;
inputting, as a second input into the decoder, discrete latent variable data associated with a first object and a second object in the environment, the discrete latent variable data representing a first action of the first object and a second action of the second object, the second action different than the first action;
receiving, from the decoder and based at least in part on the first input and the second input, output data representing a first bounding box for the first object and a second bounding box for the second object, the first bounding box including a first orientation and the second bounding box including a second orientation; and
at least one of:
performing, based at least in part on the output data, a simulation between a vehicle, the first object, and the second object; or
controlling, based at least in part on the output data, the vehicle in the environment relative to the first object and the second object.
2. The system of claim 1 , wherein a first number of channels associated with the output data from the decoder is greater than a second number of channels associated with the first input.
3. The system of claim 1 , the operations further comprising:
determining an object type associated with the first object or the second object; and
determining the output data based at least in part on the object type.
4. The system of claim 1 , wherein the discrete latent variable data is received from a diffusion model configured to receive input data, determine cross attention data between the first object and the second object, and output the discrete latent variable data based at least in part on the cross attention data.
5. The system of claim 4 , wherein the diffusion model determines a number of objects to include in the environment based at least in part on condition data.
6. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
inputting, into a decoder of a variable autoencoder, latent variable data associated with an object in an environment, the latent variable data representing a behavior of the object;
receiving, from the decoder and based at least in part on the latent variable data, output data representing a discrete occupancy representation for the object; and
at least one of:
performing, based at least in part on the output data, a simulation between a robotic device and the object: or
controlling, based at least in part on the output data, a robotic device in the environment.
7. The one or more non-transitory computer-readable media of claim 6 , wherein the latent variable data comprises discrete latent variable data representing at least one of an action, an intent, or an attribute of the object for use during the simulation, and the operations further comprising:
inputting map data representing the environment into the decoder; and
determining, by the decoder, the output data based at least in part on the map data.
8. The one or more non-transitory computer-readable media of claim 6 , wherein:
the object is a first object,
the behavior is a first behavior,
the latent variable data further represents a second behavior of a second object, and
the output data comprises a second occupancy representation.
9. The one or more non-transitory computer-readable media of claim 6 , wherein a first number of channels associated with the output data from the decoder is greater than a second number of channels associated with an input to an encoder of the variable autoencoder.
10. The one or more non-transitory computer-readable media of claim 6 , wherein the output data further represents an orientation of the object.
11. The one or more non-transitory computer-readable media of claim 6 , wherein the latent variable data is received from a diffusion model configured to receive input data and apply an algorithm to denoise the input data.
12. The one or more non-transitory computer-readable media of claim 11 , wherein the diffusion model determines a number of objects to include in the environment.
13. The one or more non-transitory computer-readable media of claim 6 , the operations further comprising:
determining, based at least in part on the output data, one or more of: a position, a size, an acceleration, or a velocity of the object at a future time.
14. The one or more non-transitory computer-readable media of claim 6 , wherein the occupancy representation for the object comprises a bounding box representing a two-dimensional shape or a three-dimensional shape of the object for a current time.
15. The one or more non-transitory computer-readable media of claim 6 , wherein the output data comprises a vector representation of the object.
16. The one or more non-transitory computer-readable media of claim 6 , wherein:
the decoder is trained based at least in part on an output from an encoder that is configured to receive map data and occupancy data associated with the object as input.
17. A method comprising:
inputting, into a decoder of a variable autoencoder, latent variable data associated with an object in an environment, the latent variable data representing a behavior of the object;
receiving, from the decoder and based at least in part on the latent variable data, output data representing a discrete occupancy representation for the object; and
at least one of:
performing, based at least in part on the output data, a simulation between a robotic device and the object; or
controlling, based at least in part on the output data, a robotic device in the environment.
18. The method of claim 17 , wherein the latent variable data comprises discrete latent variable data representing at least one of an action, an intent, or an attribute of the object for use during the simulation, and further comprising:
inputting map data representing the environment into the decoder; and
determining, by the decoder, the output data based at least in part on the map data.
19. The method of claim 17 , wherein a first number of channels associated with the output data from the decoder is greater than a second number of channels associated with an input to an encoder of the variable autoencoder.
20. The method of claim 17 , wherein the latent variable data is received from a diffusion model configured to receive input data and apply an algorithm to denoise the input data.