IP Library › Granted Patent US 12,637,112
Granted Patent B2
US 12,637,112 · App. 18/676,029 · Granted May 26, 2026

Systems and methods for generating synthetic motion predictions

Inventors: Shun Da Suo (Toronto, CA); Sebastián David Regalado Lozano (Toronto, CA); Sergio Casas (Toronto, CA); Raquel Urtasun (Toronto, CA)
Assignee: AURORA OPERATIONS, INC.
B60W60/00276G06F18/2148G06N3/045G06V20/584
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,637,112
App. No.
18/676,029
Granted
May 26, 2026
Kind
B2
Abstract

Systems and methods for generating synthetic testing data for autonomous vehicles are provided. A computing system can obtain map data descriptive of an environment and object data descriptive of a plurality of objects within the environment. The computing system can generate context data including deep or latent features extracted from the map and object data by one or more machine-learned models. The computing system can process the context data with a machine-learned model to generate synthetic motion prediction for the plurality of objects. The synthetic motion predictions for the objects can include one or more synthesized states for the objects at future times. The computing system can provide, as an output, synthetic testing data that includes the plurality of synthetic motion predictions for the objects. The synthetic testing data can be used to test an autonomous vehicle control system in a simulation.

Claims (35)

1 . A computer-implemented method for generating synthetic testing data for autonomous vehicles, the method comprising:

(a) obtaining object data descriptive of a plurality of objects within an environment, wherein the object data comprises respective current states of the plurality of objects within the environment;

(b) generating context data associated with the plurality of objects within the environment based at least in part on the object data;

(c) processing the context data with a machine-learned multi-agent behavior model to simulate traffic in the environment to generate a plurality of synthetic motion predictions respectively for the plurality of objects, wherein the plurality of synthetic motion predictions comprise one or more synthesized states for the plurality of objects, wherein the machine-learned model has been trained with a multi-task loss to jointly generate the plurality of synthetic motion predictions; and

(d) providing, as an output, synthetic testing data that includes at least a portion of the plurality of synthetic motion predictions respectively for the plurality of objects.

2 . The computer-implemented method of claim 1 , wherein the multi-task loss includes an imitation term that encourages the machine-learned model to generate the plurality of synthetic motion predictions to imitate ground truth motions.

3 . The computer-implemented method of claim 1 , wherein the machine-learned model is configured to parameterize a joint actor policy that generates plans for the plurality of objects in a scene jointly.

4 . The computer-implemented method of claim 3 , the machine-learned model having been trained by unrolling the joint actor policy for closed-loop training.

5 . The computer-implemented method of claim 4 , wherein the closed-loop training comprises determining a loss at multiple respective time steps.

6 . The computer-implemented method of claim 1 , wherein the machine-learned model comprises a generative model.

7 . The computer-implemented method of claim 1 , wherein the machine-learned model comprises an implicit latent variable model.

8 . The computer-implemented method of claim 7 , wherein the implicit latent variable model is configured to generate a latent variable distribution representing object maneuvers or interactions among the plurality of objects within the environment.

9 . The computer-implemented method of claim 8 , wherein the interactions among the plurality of objects comprise one or more of yielding actions, merging actions, or passing actions.

10 . The computer-implemented method of claim 8 , wherein the object maneuvers comprise one or more of a three-point turn, a U-turn, or a maneuver that does not comply with traffic rules.

11 . The computer-implemented method of claim 1 , further comprising:

(e) running a simulation to test an autonomous vehicle control system using the synthetic testing data, wherein during at least a portion of the simulation, a simulated object moves within a simulated environment in accordance with at least one synthetic motion prediction of the plurality of synthetic motion predictions.

12 . A computing system comprising:

one or more processors; and

one or more computer-readable medium storing instructions that when executed by the one or more processors cause the computing system to perform operations, the operations comprising:

(a) obtaining object data descriptive of a plurality of objects within an environment, wherein the object data comprises respective current states of the plurality of objects within the environment;

(b) generating context data associated with the plurality of objects within the environment based at least in part on the object data;

(c) processing the context data with a machine-learned multi-agent behavior model to simulate traffic in the environment to generate a plurality of synthetic motion predictions respectively for the plurality of objects, wherein the plurality of synthetic motion predictions comprise one or more synthesized states for the plurality of objects, wherein the machine-learned model has been trained with a multi-task loss to jointly generate the plurality of synthetic motion predictions; and

(d) providing, as an output, synthetic testing data that includes at least a portion of the plurality of synthetic motion predictions respectively for the plurality of objects.

13 . The computing system of claim 12 , wherein the multi-task loss includes an imitation term that encourages the machine-learned model to generate the plurality of synthetic motion predictions to imitate ground truth motions.

14 . The computing system of claim 12 , wherein the machine-learned model is configured to parameterize a joint actor policy that generates plans for the plurality of objects in a scene jointly.

15 . The computing system of claim 14 , the machine-learned model having been trained by unrolling the joint actor policy for closed-loop training.

16 . The computing system of claim 12 , wherein the machine-learned model comprises a generative model.

17 . The computing system of claim 12 , wherein the machine-learned model comprises an implicit latent variable model.

18 . The computing system of claim 17 , wherein the implicit latent variable model is configured to generate a latent variable distribution representing object maneuvers or interactions among the plurality of objects within the environment.

19 . The computing system of claim 12 , further comprising: (e) training one or more machine learning models of an autonomous vehicle control system via performance of machine learning algorithms on one or more training examples comprising the synthetic testing data.

20 . One or more non-transitory computer-readable media that store: one or more machine-learned models, wherein the one or more machine-learned models have been learned via performance of machine learning algorithms on one or more training examples comprising synthetic testing data, the synthetic testing data having been generated by performance of operations, the operations comprising:

(a) obtaining object data descriptive of a plurality of objects within an environment, wherein the object data comprises respective current states of the plurality of objects within the environment;

(b) generating context data associated with the plurality of objects within the environment based at least in part on the object data;

(c) processing the context data with a machine-learned multi-agent behavior model to simulate traffic in the environment to generate a plurality of synthetic motion predictions respectively for the plurality of objects, wherein the plurality of synthetic motion predictions comprise one or more synthesized states for the plurality of objects, wherein the machine-learned model has been trained with a multi-task loss to jointly generate the plurality of synthetic motion predictions; and

(d) providing, as an output, synthetic testing data that includes at least a portion of the plurality of synthetic motion predictions respectively for the plurality of objects.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 9, 2026
From: CASAS, SERGIO; SUO, SHUN DA; LOZANO, SEBASTIÁN DAVID REGALADO; URTASUN, RAQUEL
To: UATC, LLC
Reel/Frame 073416/0808 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2024
From: UATC, LLC
To: AURORA OPERATIONS, INC.
Reel/Frame 067733/0001 →
Continuity (3)
Continuation 17528577 · Nov 17, 2021
Provisional Application 63114862 · Nov 17, 2020
Related Publication 20240391504A1 · Nov 28, 2024
References Cited (48)
US 20200104685A1 · Hasenclever et al. · 2020 [cited by applicant]
US 20200125902A1 · Couse et al. · 2020 [cited by applicant]
US 20200342242A1 · Avidan · 2020 [cited by examiner]
US 20210149022A1 · Kehl · 2021 [cited by examiner]
US 20210390761A1 · Kowalski · 2021 [cited by examiner]
Alahi et al., “Social LTSM: Human Trajectory Prediction in Crowded Spaces”, IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27-30, 2016, Las Vegas, Nevada, pp. 961-971. [cited by applicant]
Bansal et al., “ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst”, arXiv:1812.03079, Dec. 7, 2018, 20 pages. [cited by applicant]
Behbahani et al., “Learning from Demonstration in the Wild”, arXiv:1811.03516v2, Mar. 26, 2019, 9 pages. [cited by applicant]
Best et al., “Autono Vi-Sim: Autonomous Vehicle Simulation Platform with Weather, Sensing, and Traffic Control”, IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 18-23, 2018, Salt Lake City, Utah, pp… [cited by applicant]
Bhattacharyya et al., “Modeling Human Driving Behavior through Generative Adversarial Imitation Learning”, arXiv:2006.06412, Jun. 10, 2020, 28 pages. [cited by applicant]
Bhattacharyya et al., “Simulating Emergent Properties of Human Driving Behavior Using Multi-Agent Reward Augmented Imitation Learning”, arXiv:1903.05766v1, Mar. 14, 2019, 7 pages. [cited by applicant]
Bhattacharyya et al., “Multi-Agent Imitation Learning for Driving Simulation”, arXiv:1903.05766v1, Mar. 14. 2019, 7 pages. [cited by applicant]
Bojarski et al., “End to End Learning for Self-Driving Cars”, arXiv:1604.07316v1, Apr. 25, 2016, 9 pages. [cited by applicant]
Brantley et al., “Disagreement-Regularized Imitation Learning”, International Conference on Learning Representations, Apr. 30, 2020, Addis Ababa, Ethiopia, pp. 1-16. [cited by applicant]
Casas et al., “SPAGNN: Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data” arXiv1910.08233, Oct. 18, 2019, 11 pages. [cited by applicant]
Casas et al., Implicit Latent Variable Model for Scene-Consistent Motion Forecasting, European Conference on Computer Vision, Aug. 23-29, 2020, pp. 624-641. [cited by applicant]
Casas et al., “The Importance of Prior Knowledge in Precise Multimodal Prediction”, arXiv:2006.0263v1, Jun. 4, 2020, 10 pages. [cited by applicant]
Casas et al., “IntentNet: Learning to Predict Intention from Raw Sensor Data”, Conference on Robot Learning, Oct. 29-31, 2018, Zurich, Switzerland, pp. 947-956. [cited by applicant]
Chai et al., “MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction”, arXiv:1910.05449v1, Oct. 12, 2019, 14 pages. [cited by applicant]
Chao et al., “A Survey on Visual Traffic Simulation: Models, Evaluations, and Applications in Autonomous Driving”, Computer Graphics Forum, vol. 39, No. 1, 2020, pp. 287-308. [cited by applicant]
Codevilla et al., “End-to-End Driving via Conditional Imitation Learning”, arXiv:1710.02410v2, Mar. 2, 2018, 8 pages. [cited by applicant]
Cui et al., “Multimodal Trajectory Predictions for Autonomous Driving Using Deep Convolutional Networks”, arXiv:1809.10732v2, Mar. 1, 2019, 7 pages. [cited by applicant]
Djuric et al., “Short-Term Motion Prediction of Traffic Actors for Autonomous Driving Using Deep Convolutional Networks”, arXiv:1808.05819v2, Sep. 16, 2018, 7 pages. [cited by applicant]
Dosovitskiy et al., “CARLA: An Open Urban Driving Simulator”, 1 [cited by applicant]
Gipps, “A Behavioural Car-Following Model for Computer Simulation”, Transportation Research Part B: Methodological, vol. 15, No. 2, 1981, pp. 105-111. [cited by applicant]
Henaff et al., “Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic”, arXiv:1901.02705v1, Jan. 8, 2019, 20 pages. [cited by applicant]
Higgins et al., “beta-vae: Learning Basic Visual Concepts with a Constrained Variational Framework”, International Conference on Learning Representations, Apr. 24-26, 2017, Toulon, France, pp. 1-22. [cited by applicant]
Ho et al., “Generative Adversarial Imitation Learning”, Conference on Neural Information Processing Systems, Dec. 5-10, 2016, Barcelona, Spain, pp. 1-9. [cited by applicant]
Kesting et al., “Calibrating Car-Following Models by Using Trajectory Data: Methodological Study”, arXiv:0803.4063v1, Mar. 28, 2008, 17 pages. [cited by applicant]
Kesting et al., “General Lane-Changing Model MOBIL for Car-Following Models”, Transportation Research Record, vol. 1999, No. 1, 2007, pp. 86-94. [cited by applicant]
Kesting et al., “Agents for Traffic Simulation”, arXiv:0805.0300vl, May 2, 2008, 34 pages. [cited by applicant]
Kingma et al., “Auto-Encoding Variational Bayes”, arXiv: 1312.6114v10, May 1, 2014, 14 pages. [cited by applicant]
Kong et al., “Kinematic and Dynamic Vehicle Models for Autonomous Driving Control Design”, IEEE Intelligent Vehicles Symposium, Jun. 28-Jul. 1, 2015, Seoul, Korea, pp. 1094-1099. [cited by applicant]
Li et al., “End-to-End Contextual Perception and Prediction with Interaction Transformer”, arXiv:2008.05927v1, Aug. 13, 2020, 8 pages. [cited by applicant]
Lopez et al., “Microscopic Traffic Simulation using SUMO”, IEEE International Conference on Intelligent Transportation Systems, Nov. 4-7, 2018, Maul, Hawaii, pp. 2575-2582. [cited by applicant]
Ma et al., “Arbitrary-Oriented Scene Text Detection via Rotation Proposals”, arXiv:1703.01086v3, Mar. 1, 2018, 11 pages. [cited by applicant]
Manivasagam et al., “LidDARsim: Realistic LiDAR Simulation by Leveraging the Real World”, Conference on Computer Vision and Pattern Recognition, Jun. 13-19, 2020, pp. 11167-11176. [cited by applicant]
Martinez et al., “Bevond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars”, arXiv:1712.01397v1, Dec. 4, 2017, 15 pages. [cited by applicant]
Pomerleau, “AL VINN: An Autonomous Land Vehicle in a Neural Network”, Advances in Neural Information Processing Systems, Nov. 30-Dec. 5, 1988, Denver, Colorado, pp. 305-313. [cited by applicant]
Rhinehart et al., “PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings”, IEEE/CVF International Conference on Computer Vision, Oct. 27-Nov. 3, 2019, Seoul, South Korea pp. 2821-2830. [cited by applicant]
Ross et al., “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”, International Conference on Artificial Intelligence and Statistics, Apr. 11-13, 2011, Ft. Lauderdale, Florida, pp.… [cited by applicant]
Sohn et al., “Learning Structured Output Representation using Deep Conditional Generative Models”, Conference on Neural Information Processing Systems, Dec. 7-12, 2015, Montreal, Canada, pp. 3483-3491. [cited by applicant]
Tang et al., “Multiple Futures Prediction”, Conference on Neural Information Processing Systems, Dec. 8-14, 2019, Vancouver, Canada, pp. 1-11. [cited by applicant]
Treiber et al., “Congested Traffic States in Empirical Observations and Microscopic Simulations”, arXiv:cond-mat/0002177v2, Aug. 30, 2000, 47 pages. [cited by applicant]
Wan et al., “The Unscented Kalman Filter for Nonlinear Estimation”, Adaptive Systems for Signal Processing, Communications, and Control Symposium, Oct. 4, 2000, Lake Louise, Canada, pp. 153-158. [cited by applicant]
Wymann et al., “TORCS: The Open Racing Car Simulator”, Mar. 12, 2015. [cited by applicant]
Yang et al., “PIXOR: Real-Time 3D Object Detection from Point Clouds”, Conference on Computer Vision and Pattern Recognition, Jun. 18-22, 2018, Salt Lake City, Utah, pp. 7652-7660. [cited by applicant]
Yuan et al., “Diverse Trajectory Forecasting with Determinantal Point Processes”, arXiv:1907.04967v2, Dec. 23, 2019, 15 pages. [cited by applicant]