IP Library › Granted Patent US 12,420,844
Granted Patent B2
US 12,420,844 · App. 18/105,132 · Granted Sep 23, 2025

Neural network trajectory prediction

Inventors: Yuxiao Chen (Newark, CA); Boris Ivanovic (Mountain View, CA); Marco Pavone (Stanford, CA)
Assignee: NVIDIA Corporation
B60W60/0027G06N3/0442G06N3/0455G06N3/0464G06N3/082B60W2554/4042B60W2556/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,420,844
App. No.
18/105,132
Granted
Sep 23, 2025
Kind
B2
Abstract

Apparatuses, systems, and techniques to generate trajectory predictions. In at least one embodiment, trajectory predictions are generated based on, for example, one or more neural networks.

Claims (69)

1. A system, comprising:

at least one processor; and

at least one memory comprising instructions that, in response to execution by the at least one processor, cause the system to at least:

generate a scene graph representing agents and interactions between the agents in an environment;

partition the scene graph to identify one or more groups of the agents;

generate one or more probability distributions indicating probabilities of one or more modes associated with the one or more groups; and

generate one or more sets of trajectories associated with the one or more groups by at least:

causing a first neural network to calculate a set of reference trajectories corresponding to a first group of agents of the one or more groups,

using a second neural network to calculate a first set of state predictions of the first group of agents based, at least in part, on the set of reference trajectories and a state of the first group of agents,

using the second neural network to calculate a second set of state predictions of the first group of agents based, at least in part, on the first set of state predictions and the state of the first group of agents, and

calculating a first set of trajectories of the one or more sets of trajectories based, at least in part, on the first set of state predictions and the second set of state predictions; and

cause a particular agent of the agents to move within the environment relative to at least one other of the agents based at least in part on at least a portion of the one or more sets of trajectories.

2. The system of claim 1 , wherein the first set of trajectories indicate predicted trajectories of the first group of agents through the environment.

3. The system of claim 1 , wherein the first neural network comprises one or more gated recurrent units (GRUs) to calculate the set of reference trajectories based, at least in part, on a history of the first group of agents.

4. The system of claim 1 , wherein the one or more probability distributions include a Gibbs distribution.

5. The system of claim 1 , wherein the first set of trajectories are in reference to one or more modes of the first group of agents in the environment.

6. The system of claim 1 , the at least one memory comprising further instructions that, in response to execution by the at least one processor, cause the system to at least:

train the first neural network and the second neural network in connection with one or more conditional value at risk (CVaR) loss functions.

7. The system of claim 1 , wherein the particular agent is a vehicle, and causing the vehicle to move within the environment relative to the at least one other of the agents based at least in part on the portion of the one or more sets of trajectories comprises:

calculating a trajectory of the vehicle in the environment based, at least in part, on the first set of trajectories; and

causing the vehicle to move in accordance with the trajectory.

8. The system of claim 1 , wherein the one or more groups of agents comprise one or more shared characteristics.

9. A method, comprising:

generating a scene graph to calculate one or more groups of agents from a plurality of agents in a scene, the scene graph representing the plurality of agents and agent interactions in the scene; and

generating a plurality of trajectory predictions associated with the one or more groups of agents by at least:

using a first neural network to generate one or more reference trajectories associated with a first group of agents of the one or more groups of agents,

using a second neural network to generate one or more action predictions corresponding to one or more agents of the first group of agents based, at least in part, on a state of the one or more agents and the one or more reference trajectories,

updating the state based, at least in part, on the one or more action predictions, and

generating at least a first portion of the plurality of trajectory predictions based, at least in part, on the updated state; and

causing one or more agents of the plurality of agents to move within the scene relative to at least one other of the plurality of agents based at least in part on at least a portion of the plurality of trajectory predictions.

10. The method of claim 9 , further comprising:

generating each action prediction of the one or more action predictions in parallel.

11. The method of claim 9 , further comprising:

generating a second portion of the plurality of trajectory predictions corresponding to a second group of agents of the one or more groups of agents, wherein the second group of agents is different from the first group of agents.

12. The method of claim 9 , further comprising:

calculating the one or more groups of agents based, at least in part, on a constant velocity model.

13. The method of claim 9 , wherein the one or more reference trajectories are based, at least in part, on one or more modes of operation of the one or more agents.

14. The method of claim 9 , further comprising:

training the first neural network and the second neural network in connection with one or more evidence lower bound (ELBO) loss functions.

15. The method of claim 9 , further comprising:

calculating one or more state predictions based, at least in part, on the one or more action predictions; and

updating the state to be in accordance with the one or more state predictions.

16. The method of claim 9 , wherein the one or more agents comprise one or more vehicles, and

causing the one or more vehicles to move within the scene relative to the at least one other of the plurality of agents based at least in part on the portion of the plurality of trajectory predictions comprises causing the one or more vehicles to navigate to avoid one or more trajectories indicated by the plurality of trajectory predictions.

17. A non-transitory computer-readable medium comprising instructions that, when performed by at least one processor of a computing device, cause the computing device to at least:

partition a scene graph into one or more groups of agents in an environment, the scene graph representing the agents and agent interaction in the environment; and

calculate one or more sets of trajectory predictions in the environment corresponding to the one or more groups by at least:

causing a first neural network to calculate a first set of trajectories associated with one or more agents of a first group of the one or more groups,

using a second neural network to calculate a first set of state predictions of the one or more agents based, at least in part, on the first set of trajectories and one or more states of the one or more agents,

updating the one or more states based, at least in part, on the first set of state predictions, and

calculating a first set of trajectory predictions based, at least in part, on the updated one or more states; and

cause one or more of the agents to move within the environment relative to at least one other of the agents based at least in part on at least a portion of the one or more sets of trajectory predictions.

18. The non-transitory computer-readable medium of claim 17 , comprising further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:

cause the first neural network to calculate a second set of trajectories associated with the one or more agents of the first group;

use the second neural network to calculate a second set of state predictions of the one or more agents based, at least in part, on the second set of trajectories and the one or more states of the one or more agents;

update the one or more states based, at least in part, on the second set of state predictions; and

calculate a second set of trajectory predictions based, at least in part, on the updated one or more states, wherein the second set of trajectory predictions are associated with a second mode different from a first mode that the first set of trajectory predictions are associated with.

19. The non-transitory computer-readable medium of claim 17 , comprising further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:

calculate one or more probability values associated with the first set of trajectory predictions.

20. The non-transitory computer-readable medium of claim 17 , wherein the one or more agents comprise one or more autonomous vehicle systems, and the non-transitory computer-readable medium comprises further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:

provide the first set of trajectory predictions to the one or more autonomous vehicle systems.

21. The non-transitory computer-readable medium of claim 17 , wherein the first set of trajectory predictions are in reference to one or more modes.

22. The non-transitory computer-readable medium of claim 17 , comprising further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:

calculate each trajectory prediction of the first set of trajectory predictions in parallel, wherein each trajectory prediction of the first set of trajectory predictions corresponds to a respective agent of the one or more agents of the first group.

23. The non-transitory computer-readable medium of claim 17 , comprising further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:

select the portion of the one or more sets of trajectory predictions based, at least in part, on one or more associated probability values.

24. The non-transitory computer-readable medium of claim 17 , comprising further instructions that when performed by the at least one processor of the computing device, cause the computing device to at least:

calculate one or more trajectories of one or more entities in the environment based, at least in part, on the one or more sets of trajectory predictions; and

cause the one or more entities to travel through the environment in accordance with the one or more trajectories.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 6, 2023
From: CHEN, YUXIAO; IVANOVIC, BORIS; PAVONE, MARCO
To: NVIDIA CORPORATION
Reel/Frame 062598/0200 →
Continuity (2)
Provisional Application 63347935 · Jun 1, 2022
Related Publication 20230391374A1 · Dec 7, 2023
References Cited (76)
US 11195418B1 · Hong · 2021 [cited by examiner]
US 11537134B1 · Wiest · 2022 [cited by examiner]
US 11816901B2 · Narayanan · 2023 [cited by examiner]
US 12001958B2 · Kamenev · 2024 [cited by examiner]
US 12110041B2 · Choi · 2024 [cited by examiner]
US 12168461B2 · Huang · 2024 [cited by examiner]
US 20190072965A1 · Zhang · 2019 [cited by examiner]
US 20190072966A1 · Zhang · 2019 [cited by examiner]
US 20190072973A1 · Sun · 2019 [cited by examiner]
US 20200110416A1 · Hong · 2020 [cited by examiner]
US 20200225669A1 · Silva · 2020 [cited by examiner]
US 20200298891A1 · Liang · 2020 [cited by examiner]
US 20200379461A1 · Singh · 2020 [cited by examiner]
US 20210009163A1 · Urtasun · 2021 [cited by examiner]
US 20210132619A1 · Refaat · 2021 [cited by examiner]
US 20210181754A1 · Cui · 2021 [cited by examiner]
US 20210192748A1 · Morales Morales · 2021 [cited by examiner]
US 20220035376A1 · Laddah · 2022 [cited by examiner]
US 20220138568A1 · Smolyanskiy · 2022 [cited by examiner]
US 20220157161A1 · Tan · 2022 [cited by examiner]
US 20220180170A1 · Ivanovic · 2022 [cited by examiner]
US 20230074293A1 · Ivanovic · 2023 [cited by examiner]
Abbeel et al., “Learning Factor Graphs in Polynomial Time and Sample Complexity,” Journal of Machine Learning Research, 2006, 46 pages. [cited by applicant]
Alahi et al., “Social Istm: Human trajectory prediction in crowded spaces,” Proceedings of the IEEE Conference an Computer Vision and Pattern Recognition 2016, pp. 961-971. [cited by applicant]
Alsterda et al., “Contingency Model Predictive Control for Automated Vehicles,” American Control Conference, 2019, 6 pages. [cited by applicant]
Bansal et al., “Chauffeurnet: Learning to Drive by Imitating the Best and Synthesizing the Worst,” 2018, 20 pages. [cited by applicant]
Batra et al., “Diverse M-Best Solutions in Markov Random Fields,” European Conference on Computer Vision, 2012, 30 pages. [cited by applicant]
Blondel et al., “Fast Unfolding of Communities in Large Networks,” Journal of Statistical Mechanics: Theory and Experiment, 2008, 12 pages. [cited by applicant]
Caesar et al., “nuScenes: A Multimodal Dataset for Autonomous Driving,” Nov. 22, 2019, 16 pages. [cited by applicant]
Casas et al., “Implicit Latent Variable Model for Scene-Consistent Motion Forecasting,” European Conference on Computer Vision, 2020, 18 pages. [cited by applicant]
Casas et al., “SPAGNN: Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data,” Oct. 18, 2019, 11 pages. [cited by applicant]
Casas et al., “The Importance of Prior Knowledge in Precise Multimodal Prediction,” International Conference on Intelligent Robots and Systems, 2020, 10 pages. [cited by applicant]
Chen et al., “Interactive Multi-Modal Motion Planning with Branch Model Predictive Control,” 2021, 10 pages. [cited by applicant]
Chen et al., “Reactive Motion Planning with Probabilistic Safety Guarantees,” 2020, 13 pages. [cited by applicant]
Chorowski et al., “Attention-Based Models for Speech Recognition,” 2015, 19 pages. [cited by applicant]
Cui et al., “LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving,” International Conference on Computer Vision, 2021, 10 pages. [cited by applicant]
Cui et al., “Multimodal Trajectory Predictions for Autonomous Driving Using Deep Convolutional Networks,” International Conference on Robotics and Automation, 2019, 7 pages. [cited by applicant]
Deo et al., “Convolutional Social Pooling for Vehicle Trajectory Prediction,” IEEE Conference on Computer Vision and Pattern Recognition, 2018, 9 pages. [cited by applicant]
Deo et al., “Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs,” IEEE Intelligent Vehicles Symposium, 2018, 6 pages. [cited by applicant]
Djuric et al., “MultiXNet: Multiclass Multistage Multimodal Motion Prediction,” 2021, 8 pages. [cited by applicant]
Dubins, “On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents,” American Journal of mathematics, 79(3): Jul. 1957, 20 pages. [cited by applicant]
Gupta et al., “Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks,” IEEE Conference on Computer Vision and Pattern Recognition, 2018, 10 pages. [cited by applicant]
Helbing et al., “Social Force Model for Pedestrian Dynamics,” Physical Review, 51(5): 1995, 18 pages. [cited by applicant]
Huang et al., “DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling,” IEEE Robotics and Automation Letters, 5(4): 2020, 8 pages. [cited by applicant]
IEEE, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages. [cited by applicant]
Ivanovic et al., “MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control,” Conference on Robot Learning, 2020, 14 pages. [cited by applicant]
Ivanovic et al., “The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs,” IEEE/CVF International Conference on Computer Vision, 2019, 10 pages. [cited by applicant]
Kamenev et al., “PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation,” 2021, 7 pages. [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201609, issued Jan… [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201806, issued Jan… [cited by applicant]
Kitani et al., “Activity Forecasting,” European Conference on Computer Vision, 2012, 14 pages. [cited by applicant]
Kuefler et al., “Imitating Driver Behavior with Generative Adversarial Networks,” IEEE Intelligent Vehicles Symposium, 2017, 8 pages. [cited by applicant]
Lee et al., “DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents,” IEEE Conference on Computer Vision and Pattern Recognition, 2017, 10 pages. [cited by applicant]
Lerner et al., “Crowds by example,” Computer Graphics Forum, vol. 26, 2007, 10 pages. [cited by applicant]
Liu et al., “Multimodal Motion Prediction with Stacked Transformers,” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, 10 pages. [cited by applicant]
Ma et al., “Forecasting Interactive Dynamics of Pedestrians with Fictitious Play,” IEEE Conference on Computer Vision and Pattern Recognition, 2017, 9 pages. [cited by applicant]
Mehran et al., “Abnormal crowd behavior detection using social force model,” IEEE Conference on Computer Vision and Pattern Recognition, 2009, 8 pages. [cited by applicant]
Ngiam et al., “Scene Transformer: A Unified Multi-task Model for Behavior Prediction and Planning,” 2021, 21 pages. [cited by applicant]
Pellegrini et al., “You'll Never Walk Alone: Modeling Social Behavior for Multi-target Tracking,” IEEE 12th International Conference on Computer Vision, 2019, 8 pages. [cited by applicant]
Phan-Minh et al., “CoverNet: Multimodal Behavior Prediction using Trajectory Sets,” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, 10 pages. [cited by applicant]
Ranney et al., “Models of Driving Behavior: A Review of Their Evolution,” Accident Analysis & Prevention, 26(6): 1994, 18 pages. [cited by applicant]
Rhinehart et al., “PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings,” IEEE/CVF International Conference on Computer Vision, 2019, 10 pages. [cited by applicant]
Rockafellar et al., “Optimization of Conditional Value-at-Risk,” Journal of risk, 2:21-42, 2000, 21 pages. [cited by applicant]
Sadeghian et al., “SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints,” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, 10 pages. [cited by applicant]
Sadeghian et al., “CAR-Net: Clairvoyant Attentive Recurrent Network,” European Conference on Computer Vision, 2018, 17 pages. [cited by applicant]
Salzmann et al., “Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data,” European Conference on Computer Vision, 2020, 17 pages. [cited by applicant]
Sohn et al., “Learning Structured Output Representation using Deep Conditional Generative Models,” Advances in heural information processing systems, 2015, 9 pages. [cited by applicant]
Tang et al., “Multiple futures prediction,” Advances in Neural Information Processing Systems, 2019, 11 pages. [cited by applicant]
Treiber et al., “Congested Traffic States in Empirical Observations and Microscopic Simulations,” Physical review, 62(2):2000, 49 pages. [cited by applicant]
Vemula et al., “Social Attention: Modeling Attention in Human Crowds,” International Conference on Robotics and Automation, 2018, 2018, 7 pages. [cited by applicant]
Wachter et al., “On the Implementation of an Interior-Point Filter Line-Search Algorithm for Largescale Nonlinear Programming,” Mathematical Programming, 106(1): 2006, 33 pages. [cited by applicant]
Weng et al., “PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling,” IEEE Robotics and Automation Letters, 6(3): 2021, 8 pages. [cited by applicant]
Yuan et al., “Diverse Trajectory Forecasting with Determinantal Point Processes,” 2019, 15 pages. [cited by applicant]
Yuan et al., “AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting,” IEEE/CVF International Conference on Computer Vision, 2021, 14 pages. [cited by applicant]
Zhao et al., “Multi-Agent Tensor Fusion for Contextual Trajectory Prediction,” IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, 9 pages. [cited by applicant]
Zhou et al., “Grouptron: Dynamic Multi-Scale Graph Convolutional Networks for Group-Aware Dense Crowd Trajectory Forecasting,” 2021, 7 pages. [cited by applicant]
Cited By (2)
US 12,565,215 US 12,654,320