IP Library › Granted Patent US 12,626,589
Granted Patent B2
US 12,626,589 · App. 18/168,093 · Granted May 12, 2026

Systems and methods for simulating traffic scenes

Inventors: Shuhan Tan (Austin, TX); Kelvin Ka Wing Wong (Vancouver, CA); Shenlong Wang (Urbana, IL); Sivabalan Manivasagam (Toronto, CA); Mengye Ren (Toronto, CA); Raquel Urtasun (Toronto, CA)
Assignee: AURORA OPERATIONS, INC.
G08G1/0133G06F18/2415G06V20/54G08G1/0129
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,626,589
App. No.
18/168,093
Granted
May 12, 2026
Kind
B2
Abstract

Example aspects of the present disclosure describe a scene generator for simulating scenes in an environment. For example, snapshots of simulated traffic scenes can be generated by sampling a joint probability distribution trained on real-world traffic scenes. In some implementations, samples of the joint probability distribution can be obtained by sampling a plurality of factorized probability distributions for a plurality of objects for sequential insertion into the scene.

Claims (50)

1 . A computer-implemented method for traffic scene generation, comprising:

(a) obtaining environmental data descriptive of an environment and a subject vehicle within the environment;

(b) generating a first parameter of a new object for insertion into a synthesized traffic scene in the environment, the first parameter generated using a first parameter prediction machine-learned model of a machine-learned traffic scene generation framework and based at least in part on the environmental data;

(c) generating a second parameter of the new object, the second parameter generated using a second parameter prediction machine-learned model of the machine-learned traffic scene generation framework and based at least in part on the environmental data and the first parameter; and

(d) outputting data descriptive of the synthesized traffic scene.

2 . The computer-implemented method of claim 1 , further comprising:

generating feature data using a machine-learned feature extraction model based on a current state of the synthesized traffic scene.

3 . The computer-implemented method of claim 2 , wherein the machine-learned feature extraction model processes multiple prior states of the synthesized traffic scene.

4 . The computer-implemented method of claim 2 , wherein the first parameter and the feature data are provided as inputs to the second parameter prediction machine-learned model.

5 . The computer-implemented method of claim 4 , further comprising:

(e) inputting the first parameter, the second parameter, and the feature data to a third parameter prediction machine-learned model of the machine-learned traffic scene generation framework.

6 . The computer-implemented method of claim 1 , wherein at least one of the first parameter or the second parameter describes at least one of: an object class, an object position, an object orientation, an object bounding box, or an object velocity.

7 . The computer-implemented method of claim 1 , further comprising:

(f) generating simulated sensor data for the environment based on the data descriptive of the synthesized traffic scene;

(g) obtaining labels for the simulated sensor data based on the first parameter and the second parameter; and

(h) training a machine-learned model of an autonomous vehicle control system using the labels and the simulated sensor data.

8 . A computing system for traffic scene generation, the computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the computing system to perform operations, the operations comprising:

(a) obtaining environmental data descriptive of an environment and a subject vehicle within the environment;

(b) generating a first parameter of a new object for insertion into a synthesized traffic scene in the environment, the first parameter generated using a first parameter prediction machine-learned model of a machine-learned traffic scene generation framework and based at least in part on the environmental data;

(c) generating a second parameter of the new object, the second parameter generated using a second parameter prediction machine-learned model of the machine-learned traffic scene generation framework and based at least in part on the environmental data and the first parameter; and

(d) outputting data descriptive of the synthesized traffic scene.

9 . The computing system of claim 8 , further comprising:

generating feature data using a machine-learned feature extraction model based on a current state of the synthesized traffic scene.

10 . The computing system of claim 9 , wherein the machine-learned feature extraction model processes multiple prior states of the synthesized traffic scene.

11 . The computing system of claim 9 , wherein the first parameter and the feature data are provided as inputs to the second parameter prediction machine-learned model.

12 . The computing system of claim 11 , further comprising:

(e) inputting the first parameter, the second parameter, and the feature data to a third parameter prediction machine-learned model of the machine-learned traffic scene generation framework.

13 . The computing system of claim 8 , wherein at least one of the first parameter or the second parameter describes at least one of: an object class, an object position, an object orientation, an object bounding box, or an object velocity.

14 . The computing system of claim 8 , wherein the operations comprise:

(f) generating simulated sensor data for the environment based on the data descriptive of the synthesized traffic scene;

(g) obtaining labels for the simulated sensor data based on the first parameter and the second parameter; and

(h) training a machine-learned model of an autonomous vehicle control system using the labels and the simulated sensor data.

15 . An autonomous vehicle control system comprising:

one or more machine-learned models that have been trained using simulated sensor data representing at least a portion of a synthesized traffic scene, the simulated sensor data having been generated by performance of operations, the operations comprising:

(a) obtaining environmental data descriptive of an environment and a subject vehicle within the environment;

(b) generating a first parameter of a new object for insertion into a synthesized traffic scene in the environment, the first parameter generated using a first parameter prediction machine-learned model of a machine-learned traffic scene generation framework and based at least in part on the environmental data;

(c) generating a second parameter of the new object, the second parameter generated using a second parameter prediction machine-learned model of the machine-learned traffic scene generation framework and based at least in part on the environmental data and the first parameter; and

(d) outputting data descriptive of the synthesized traffic scene.

16 . The autonomous vehicle control system of claim 15 , the operations further comprising:

generating feature data using a machine-learned feature extraction model based on a current state of the synthesized traffic scene.

17 . The autonomous vehicle control system of claim 16 , wherein the machine-learned feature extraction model processes multiple prior states of the synthesized traffic scene.

18 . The autonomous vehicle control system of claim 16 , wherein the first parameter and the feature data are provided as inputs to the second parameter prediction machine-learned model.

19 . The autonomous vehicle control system of claim 18 , the operations further comprising:

(e) inputting the first parameter, the second parameter, and the feature data to a third parameter prediction machine-learned model of the machine-learned traffic scene generation framework.

20 . The autonomous vehicle control system of claim 15 , the operations further comprising:

(f) generating simulated sensor data for the environment based on the data descriptive of the synthesized traffic scene;

(g) obtaining labels for the simulated sensor data based on the first parameter and the second parameter; and

(h) training at least one of the one or more machine-learned models of the autonomous vehicle control system using the labels and the simulated sensor data.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2024
From: UATC, LLC
To: AURORA OPERATIONS, INC.
Reel/Frame 067733/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 23, 2023
From: WONG, KELVIN KA WING; TAN, SHUHAN; WANG, SHENLONG; REN, MENGYE; MANIVASAGAM, SIVABALAN; URTASUN, RAQUEL
To: UATC, LLC
Reel/Frame 062783/0383 →
Continuity (3)
Continuation 17528277 · Nov 17, 2021
Provisional Application 63114848 · Nov 17, 2020
Related Publication 20230196909A1 · Jun 22, 2023
References Cited (67)
US 20200134494A1 · Venkatadri · 2020 [cited by applicant]
US 20200302161A1 · Sriram et al. · 2020 [cited by applicant]
US 20200342242A1 · Avidan · 2020 [cited by examiner]
US 20200363800A1 · Jojo-Verge et al. · 2020 [cited by applicant]
US 20200410062A1 · O'Malley · 2020 [cited by applicant]
US 20200410063A1 · O'Malley · 2020 [cited by applicant]
US 20210049243A1 · Venkatadri · 2021 [cited by applicant]
US 20210275918A1 · Devaranjan et al. · 2021 [cited by applicant]
US 20210279640A1 · Tu et al. · 2021 [cited by applicant]
US 20210312244A1 · Atsmon et al. · 2021 [cited by applicant]
US 20210390761A1 · Kowalski · 2021 [cited by examiner]
US 20220101074A1 · Peters et al. · 2022 [cited by applicant]
Bando et al., “Dynamical model of traffic congestion and numerical simulation.” Physical review E 51, No. 2, 1995, p. 1035. [cited by applicant]
Bengio et al., “Modeling high-dimensional discrete data with multi-layer neural networks.” Advances in Neural Information Processing Systems 12, 1999, 7 pages. [cited by applicant]
Best et al., “Efficient simulation of the von Mises distribution.” Journal of the Royal Statistical Society: Series C (Applied Statistics) 28, No. 2, 1979, pp. 152-157. [cited by applicant]
Binkowski et al., “Demystifying mmd gans.” ar Xiv preprint arXiv:1801.01401, 2018, 36 pages. [cited by applicant]
Bishop et al., “Pattern recognition and machine learning”. vol. 4, No. 4. New York: springer, 2006, 11 pages. [cited by applicant]
Chandler et al., “Traffic dynamics: studies in car following.” Operations research 6, No. 2, 1958, pp. 165-184. [cited by applicant]
Chang et al., “Argoverse: 3d tracking and forecasting with rich maps.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8748-8757. [cited by applicant]
Dosovitskiy et al., “Carla: An open urban driving simulator.” In Conference on robot learning, PMLR. 2017, pp. 1-16. [cited by applicant]
Fang et al., “Augmented lidar simulator for autonomous driving.” IEEE Robotics and Automation Letters 5, No. 2, 2020, pp. 1931-1938. [cited by applicant]
Fellendorf, “Vissim: A microscopic simulation tool to evaluate actuated signal control including bus priority.” In 64th Institute of Transportation Engineers Annual Meeting, vol. 32, Springer, 1994, pp. 1-9. [cited by applicant]
Frey et al., “Does the wake-sleep algorithm produce good density estimators?.” Advances in neural information processing systems 8, 1995, 10 pages. [cited by applicant]
Gazis et al., “Nonlinear follow-the-leader models of traffic flow.” Operations research 9, No. 4, 1961, pp. 545-567. [cited by applicant]
Geiger et al., “A generative model for 3d urban scene understanding from movable platforms.” In CVPR 2011, IEEE, 2011, pp. 1945-1952. [cited by applicant]
Geiger et al., “Joint 3d estimation of objects and scene layout.” Advances in Neural Information Processing Systems 24, 2011. [cited by applicant]
Gerlough, “Simulation of freeway traffic on a general-purpose discrete variable computer.” PhD diss., University of California, Los Angeles, 1955. [cited by applicant]
Gipps, “Computer program multsim for simulating output from vehicle detectors on a multi-lane signal-controlled road”. No. Monograph. 1976. [cited by applicant]
Graves, “Generating sequences with recurrent neural networks.” arXiv preprint arXiv:1308.0850, 2013, 43 pages. [cited by applicant]
Gretton et al., “A kernel two-sample test.” The Journal of Machine Learning Research 13, No. 1, 2012, pp. 723-773. [cited by applicant]
Ha et al., “A neural representation of sketch drawings.” arXiv preprint arXiv:1704.03477, 2017, 15 pages. [cited by applicant]
Heusel et al., “GANs trained by a two time-scale update rule converge to a local nash equilibrium.” Advances in neural information processing systems 30, 2017, 12 pages. [cited by applicant]
Hochreiter et al., “Long short-term memory.” Neural computation 9, No. 8, 1997, pp. 1735-1780. [cited by applicant]
Holtzman et al., “The curious case of neural text degeneration.” arXiv preprint arXiv:1904.09751, 2019, 16 pages. [cited by applicant]
Jesenski et al., “Generation of scenes in intersections for the validation of highly automated driving functions.” In 2019 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2019, pp. 502-509. [cited by applicant]
Jyothi et al., “LayoutVAE: Stochastic scene layout generation from a label set.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9895-9904. [cited by applicant]
Kar et al., “Meta-sim: Learning to generate synthetic datasets.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4551-4560. [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980, 2014, 15 pages. [cited by applicant]
Liang et al., “Learning lane graph representations for motion forecasting.” In European Conference on Computer Vision, Springer, Cham. 2020, pp. 541-556. [cited by applicant]
Liao et al., “Efficient graph generation with graph recurrent attention networks.” Advances in Neural Information Processing Systems 32, 2019, 11 pages. [cited by applicant]
Lighthill et al., “On kinematic waves II. A theory of traffic flow on long crowded roads.” Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 229, No. 1178, 1955, pp. 317-345. [cited by applicant]
Lopez et al., “Microscopic traffic simulation using sumo.” In 2018 21st international conference on intelligent transportation systems (ITSC), IEEE, 2018, pp. 2575-2582. [cited by applicant]
Manivasagam et al., “LIDARsim: Realistic lidar simulation by leveraging the real world.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11167-11176. [cited by applicant]
Nash et al., “Polygen: An autoregressive generative model of 3d meshes.” In International Conference on Machine Learning, PMLR, 2020, pp. 7220-7229. [cited by applicant]
Newell, “Nonlinear effects in the dynamics of car following.” Operations research 9, No. 2, 1961, pp. 209-229. [cited by applicant]
Oord et al., “Wavenet: A generative model for raw audio.” arXiv preprint arXiv:1609.03499, 2016, 15 pages. [cited by applicant]
Oord et al., “Pixel recurrent neural networks.” In International conference on machine leaming, PMLR. 2016, pp. 1747-1756. [cited by applicant]
Owen et al., “Traffic flow simulation using CORSIM.” In 2000 Winter Simulation Conference Proceedings (Cat. No. 00CH37165), vol. 2, IEEE, 2000, pp. 1143-1147. [cited by applicant]
Pipes, “An operational analysis of traffic dynamics.” Journal of applied physics 24, No. 3, 1953, pp. 274-281. [cited by applicant]
Prakash et al., “Structured domain randomization: Bridging the reality gap by context-aware synthetic data.” In 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 7249-7255. [cited by applicant]
Prokudin et al., “Deep directional statistics: Pose estimation with uncertainty quantification.” In Proceedings of the European conference on computer vision (ECCV), 2018, pp. 534-551. [cited by applicant]
Radford et al., “Language models are unsupervised multitask learners.” OpenAI blog 1, No. 8, 2019, pp. 9. [cited by applicant]
Richards, “Shock waves on the highway.” Operations research 4, No. 1, 1956, pp. 42-51. [cited by applicant]
Salimans et al., “Improved techniques for training GANs.” Advances in neural information processing systems 29, 2016, 9 pages. [cited by applicant]
Shi et al., “Convolutional LSTM network: A machine learning approach for precipitation nowcasting.” Advances in neural information processing systems 28, 2015, 9 pages. [cited by applicant]
Taheri, “An investigation and design of slip control braking systems integrated with four-wheel steering.” PhD diss., Clemson University, 1990, 274 pages. [cited by applicant]
Treiber et al., “Congested traffic states in empirical observations and microscopic simulations.” Physical review E 62, No. 2, 2000, pp. 1805. [cited by applicant]
Uria et al., “A deep and tractable density estimator.” In International Conference on Machine Learning, PMLR, 2014, pp. 467-475. [cited by applicant]
Uria et al., “Rnade: The real-valued neural autoregressive density-estimator.” Advances in Neural Information Processing Systems 26, 2013, 9 pages. [cited by applicant]
Wang et al., “Deep convolutional priors for indoor scene synthesis.” ACM Transactions on Graphics (TOG) 37, No. 4. 2018, pp. 1-14. [cited by applicant]
Wheeler et al., “Factor graph scene distributions for automotive safety analysis.” In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2016, pp. 1035-1040. [cited by applicant]
Wheeler et al., “Initial scene configurations for highway traffic propagation.” In 2015 IEEE 18th International Conference on Intelligent Transportation Systems, IEEE, 2015, pp. 279-284. [cited by applicant]
Wong et al., “Testing the safety of self-driving vehicles by simulating perception and prediction.” In European Conference on Computer Vision, Springer, Cham, 2020, pp. 312-329. [cited by applicant]
Yang et al., “A microscopic traffic simulator for evaluation of dynamic traffic management systems.” Transportation Research Part C: Emerging Technologies 4, No. 3, 1996, pp. 113-129. [cited by applicant]
Yang et al., “HDNET: Exploiting HD maps for 3D object detection.” In Conference on Robot Learning, PMLR, 2018, pp. 146-155. [cited by applicant]
You et al., “GraphRNN: Generating realistic graphs with deep auto-regressive models.” In International conference on machine learning, PMLR, 2018, pp. 5708-5717. [cited by applicant]
Zhang et al., “Understanding high-level semantics by modeling traffic patterns.” In Proceedings of the IEEE international conference on computer vision, 2013, pp. 3056-3063. [cited by applicant]