Systems and methods for simulating traffic scenes
Example aspects of the present disclosure describe a scene generator for simulating scenes in an environment. For example, snapshots of simulated traffic scenes can be generated by sampling a joint probability distribution trained on real-world traffic scenes. In some implementations, samples of the joint probability distribution can be obtained by sampling a plurality of factorized probability distributions for a plurality of objects for sequential insertion into the scene.
1. A computer-implemented method for traffic scene generation, comprising:
(a) obtaining environmental data descriptive of an environment, a subject vehicle within the environment, and an object set within the environment;
(b) sampling a plurality of parameters of a new object, wherein the plurality of parameters are sampled respectively from a plurality of probability distributions sequentially generated by a machine-learned traffic scene generation model and based at least in part on the environmental data, at least one of the plurality of probability distributions being conditioned upon one or more of the plurality of probability distributions that were previously sequentially generated;
(c) updating the environmental data by adding the new object to the object set; and
(d) outputting the environmental data descriptive of a synthesized traffic scene.
2. The computer-implemented method of claim 1 , further comprising:
iteratively performing (b) and (c) for a plurality of iterations to obtain a plurality of new objects,
wherein each of the plurality of new objects is obtained based at least in part on the environmental data updated by a prior iteration of the plurality of iterations.
3. The computer-implemented method of claim 2 , wherein, for a final iteration of the plurality of iterations, the plurality of parameters comprise an end token that, when sampled, results in termination of the plurality of iterations.
4. The computer-implemented method of claim 1 , wherein the machine-learned traffic scene generation model is configured to determine a joint probability distribution of the synthesized traffic scene over the object set conditioned on the subject vehicle.
5. The computer-implemented method of claim 4 , wherein the joint probability distribution is autoregressively factorized and comprises the plurality of probability distributions.
6. The computer-implemented method of claim 4 , wherein the machine-learned traffic scene generation model has been trained by optimizing a likelihood of real-world traffic scenes contained in a training dataset.
7. The computer-implemented method of claim 1 , wherein the machine-learned traffic scene generation model comprises:
a shared backbone feature extraction portion that extracts features from the environmental data; and
a plurality of prediction models that respectively generate the plurality of probability distributions based at least in part on the features.
8. The computer-implemented method of claim 1 , wherein the environmental data comprises a collection of polygons and polylines that provide semantic priors for a region of interest around the subject vehicle.
9. The computer-implemented method of claim 1 , wherein the environmental data comprises a multi-channel image encoding of a top-down view of the environment.
10. The computer-implemented method of claim 1 , wherein the plurality of parameters comprise one or more of: an object class, an object position, an object bounding box, or an object velocity.
11. The computer-implemented method of claim 1 , wherein the machine-learned traffic scene generation model comprises an autoregressive model comprising a convolutional long short-term memory neural network.
12. The computer-implemented method of claim 1 , further comprising:
(e) generating simulated sensor data for the environment based on the environmental data output at (d);
(f) obtaining labels for the simulated sensor data that correspond to the plurality of parameters; and
(g) training one or more machine-learned models of an autonomous vehicle control system using the labels and the simulated sensor data.
13. A computing system for traffic scene generation, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
(a) obtaining environmental data descriptive of an environment, a subject vehicle within the environment, and an object set within the environment;
(b) for one or more iterations, sampling a plurality of parameters of a new object, wherein the plurality of parameters are sampled respectively from a plurality of probability distributions sequentially generated by a machine-learned traffic scene generation model and based at least in part on the environmental data, at least one of the plurality of probability distributions being conditioned upon one or more of the plurality of probability distributions that were previously sequentially generated;
(c) for each of the one or more iterations, updating the environmental data by adding the new object to the object set; and
(d) after the one or more iterations, providing, as an output, the environmental data descriptive of a synthesized traffic scene.
14. The computing system of claim 13 , wherein the machine-learned traffic scene generation model is configured to determine a joint probability distribution of the synthesized traffic scene over the object set conditioned on the subject vehicle.
15. The computing system of claim 13 , wherein, for a final iteration of the one or more iterations, the plurality of parameters comprise an end token that, when sampled, results in termination of the one or more iterations.
16. The computing system of claim 13 , wherein the machine-learned traffic scene generation model comprises:
a shared backbone feature extraction portion that extracts features from the environmental data; and
a plurality of prediction models that respectively generate the plurality of probability distributions based at least in part on the features.
17. The computing system of claim 13 , wherein the operations further comprise:
(e) generating simulated sensor data for the environment based on the environmental data output at (d);
(f) obtaining labels for the simulated sensor data that correspond to the plurality of parameters; and
(g) training one or more machine-learned models of an autonomous vehicle control system using the labels and the simulated sensor data.
18. An autonomous vehicle control system comprising:
one or more machine-learned models that have been trained using simulated sensor data representing at least a portion of a synthesized traffic scene, the simulated sensor data having been generated by performance of operations, the operations comprising:
(a) obtaining environmental data descriptive of an environment, a subject vehicle within the environment, and an object set within the environment;
(b) for one or more iterations, sampling a plurality of parameters of a new object, wherein the plurality of parameters are sampled respectively from a plurality of probability distributions sequentially generated by a machine-learned traffic scene generation model and based at least in part on the environmental data, at least one of the plurality of probability distributions being conditioned upon one or more of the plurality of probability distributions that were previously sequentially generated;
(c) for each of the one or more iterations, updating the environmental data by adding the new object to the object set; and
(d) generating the simulated sensor data based on the environmental data updated at (c).
19. The autonomous vehicle control system of claim 18 , wherein the machine-learned traffic scene generation model is configured to determine a joint probability distribution of the synthesized traffic scene over the object set conditioned on the subject vehicle.
20. The autonomous vehicle control system of claim 19 , wherein the machine-learned traffic scene generation model comprises:
a shared backbone feature extraction portion that extracts features from the environmental data; and
a plurality of prediction models that respectively generate the plurality of probability distributions based at least in part on the features, the plurality of probability distributions autoregressively factorizing the joint probability distribution.