IP Library › Granted Patent US 12,397,823
Granted Patent B2
US 12,397,823 · App. 18/318,233 · Granted Aug 26, 2025

Differentiable and modular prediction and planning for autonomous machines

Inventors: Peter Karkus (Zurich, CH); Boris Ivanovic (Mountain View, CA); Shie Mannor (Haifa, IL); Marco Pavone (Stanford, CA)
Assignee: NVIDIA Corporation
B60W60/0011B60W30/0956B60W50/0097G06N3/08B60W2554/4041B60W2554/4045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,397,823
App. No.
18/318,233
Granted
Aug 26, 2025
Kind
B2
Abstract

In various examples, a motion planner include an analytical function to predict motion plans for a machine based on predicted trajectories of actors in an environment, where the predictions are differentiable with respect to parameters of a neural network of a motion predictor used to predict the trajectories. The analytical function may be used to determine candidate trajectories for the machine based on a predicted trajectory, to compute cost values for the candidate trajectories, and to select a reference trajectory from the candidate trajectories. For differentiability, a term of the analytical function may correspond to the predicted trajectory. A motion controller may use the reference trajectory to predict a control sequence for the machine using an analytical function trained to generate predictions that are differentiable with respect to at least one parameter of the analytical function used to compute the cost values.

Claims (75)

1. A system comprising:

one or more processing units to perform operations including:

determining a future location of an agent in an environment using a neural network having parameters trained to predict the future location of the agent in the environment;

determining a plurality of candidate trajectories for a machine based at least on the future location of the agent in the environment;

computing cost values for the plurality of candidate trajectories using one or more analytical functions trained to generate predictions corresponding to the cost values using at least one input term derived from the future location, the cost values being differentiable with respect to the parameters of the neural network through the at least one input term;

selecting a trajectory from the plurality of candidate trajectories based at least on the cost values; and

performing one or more control operations for the machine using the trajectory.

2. The system of claim 1 , wherein the operations further include:

based at least on the cost values, computing, using the trajectory, a control sequence for the machine using one or more second analytical functions trained to generate second predictions corresponding to the control sequence, the second predictions being differentiable with respect to at least one parameter of the one or more analytical functions through at least one second input term of the one or more second analytical functions.

3. The system of claim 2 , wherein the operations further include jointly training the one or more analytical functions, the one or more second analytical functions, and the neural network.

4. The system of claim 1 , wherein the determining the plurality of candidate trajectories includes:

sampling a current state of the machine in a state space and a set of terminal states for the machine to determine the plurality of candidate trajectories for the machine.

5. The system of claim 1 , wherein the selecting the trajectory from the plurality of candidate trajectories includes classifying the trajectory as a target trajectory for the machine based at least on a categorical distribution corresponding to the plurality of candidate trajectories.

6. The system of claim 1 , wherein the at least one input term computes a cost corresponding to the future location of the agent predicted using the neural network, the cost weighted using a corresponding weight value to compute a cost value of the cost values.

7. The system of claim 1 , wherein the system is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

8. The system of claim 1 , the determining the future location includes applying, to the neural network, one or more initial locations of the agent, the one or more initial locations determined using sensor data generated using one or more sensors associated with the machine.

9. A method comprising:

determining first predictions corresponding to one or more future states of one or more agents in an environment using at least one neural network having parameters trained to generate the first predictions;

computing one or more cost values for one or more candidate movements for a machine using one or more analytical functions having at least one input term derived from the one or more future states, the at least one input term including at least one parameter trained to generate second predictions corresponding to the one or more cost values, the one or more cost values being differentiable with respect to the parameters of the at least one neural network through that the at least one input term; and

performing one or more control operations for the machine based at least on the one or more cost values.

10. The method of claim 9 , wherein the method further includes:

based at least on the one or more cost values, computing, using the one or more candidate movements, a control sequence for the machine using one or more second analytical functions trained to generate third predictions corresponding to the control sequence, the third predictions being differentiable with respect to at least one parameter of the one or more analytical functions, wherein the one or more control operations correspond to the control sequence.

11. The method of claim 9 , further comprising jointly training the one or more analytical functions and the at least one neural network based at least on backpropagating gradients through the at least one input term to the parameters of the at least one neural network.

12. The method of claim 9 , wherein the one or more candidate movements are determined based at least on:

sampling a current state of the machine in a state space and a set of terminal states for the machine to determine the one or more candidate movements for the machine.

13. The method of claim 9 , wherein the performing the one or more control operations for the machine is based at least on:

classifying a candidate movement of the one or more candidate movements as a target candidate movement for the machine based at least on a categorical distribution corresponding to the one or more candidate movements.

14. The method of claim 9 , wherein the at least one input term computes a cost corresponding to the first predictions generated using the at least one neural network.

15. The method of claim 9 , wherein the at least one input term includes a Gaussian radial basis function.

16. The method of claim 9 , wherein the one or more cost values are based at least on one or more of:

one or more predicted collisions between the machine and at least one agent of the one or more agents;

one or more distances between at least one state of the machine and at least one goal state for the machine;

one or more lateral lane deviation amounts;

one or more lane heading deviation amounts; or

one or more control effort amounts.

17. The method of claim 9 , wherein the determining the first predictions includes applying, to the at least one neural network, one or more initial states of the one or more agents, the one or more initial states determined using sensor data generated using one or more sensors associated with the machine.

18. The method of claim 17 , further including generating, using at least one second neural network, third predictions corresponding to the one or more initial states of the one or more agents in the environment.

19. A processor comprising:

one or more circuits to perform one or more control operations for a machine based at least on one or more cost values determined for one or more candidate movements for the machine,

the one or more cost values determined using one or more analytical functions trained to generate first predictions corresponding to the one or more cost values using at least one input term derived from one or more future states of one or more agents,

the first predictions being differentiable with parameters of at least one neural network used to generate second predictions corresponding to the one or more future states through the at least one input term.

20. The processor of claim 19 , wherein the one or more circuits are further to:

based at least on the one or more cost values, compute, using the one or more candidate movements, a control sequence for the machine using one or more second analytical functions trained to generate third predictions corresponding to the control sequence, the third predictions being differentiable with respect to at least one parameter of the one or more analytical functions, wherein the one or more control operations correspond to the control sequence.

21. The processor of claim 19 , wherein the one or more analytical functions and the at least one neural network are jointly trained.

22. The processor of claim 19 , wherein the processor is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 30, 2023
From: KARKUS, PETER; IVANOVIC, BORIS; MANNOR, SHIE; PAVONE, MARCO
To: NVIDIA CORPORATION
Reel/Frame 063792/0292 →
Continuity (2)
Provisional Application 63359407 · Jul 8, 2022
Related Publication 20240010232A1 · Jan 11, 2024
References Cited (67)
US 20140067206A1 · Pflug · 2014 [cited by examiner]
US 20200156631A1 · Lin · 2020 [cited by examiner]
US 20200331476A1 · Chen · 2020 [cited by examiner]
US 20220340173A1 · Brown · 2022 [cited by examiner]
CN 105806353A · 2016 [cited by examiner]
DE 102020207897A1 · 2021 [cited by examiner]
Waymo. Safety report, 2021.Available at https://waymo.com/safety/safety-report. Retrieved on Jul. 4, 2021. [cited by applicant]
Argo AI. Developing a self-driving system you can trust, Apr. 2021. Available at https://www.argo.ai/wp-content/uploads/2021/04/AargoSafetyReport.pdf.; 55 pgs. [cited by applicant]
Jatavallabhula, et al.; “GradSLAM: Dense SLAM Meets Automatic Differentiation”; arXiv:1910.10672, Oct. 23, 2019, 12 pgs. [cited by applicant]
Bojarski, Mariusz; “End to End Learning for Self-Driving Cars”, https://arxiv.org/abs/1604.07316; Apr. 25, 2016, 9 pgs. [cited by applicant]
Caesar, H., et al.: “NuScenes: A Multimodal Dataset for Autonomous Driving”; https://arxiv.org/abs/1903.11027; May 5, 2020, 16 pgs. [cited by applicant]
Xu, et al.; “BITS: Bi-Level Imitation for Traffic Simulation,” arXiv preprint arXiv:2208.12403, 2022. [cited by applicant]
Salzmann, et al.; “Trajectron++:Dynamically-feasible Trajectory Forecasting with Heterogeneous Data”; In European Conference on Computer Vision, pp. 683-700, Springer, 2020. [cited by applicant]
General Motors. Self-Driving Safety Report, 2018. Available at https://www.gm.com/content/dam/company/docs/us/en/gmcom/gmsafetyreport.pdf.; 33 pgs. [cited by applicant]
Uber Advanced Technologies Group. A principled approach to safety, 2020. Available at https://uber.app.box.com/v/UberATGSafetyReport; 89 pgs. [cited by applicant]
Motional. Voluntary safety self-assessment, 2021. Available at https://drive.google.com/file/d/1JjfQByU_hWvSfkWzQ8PK2ZOZfVCqQGDB/view; 54 pgs. [cited by applicant]
Zoox. Safety report vol. 2.0, 2021. Available at https://zoox.com/safety/. [cited by applicant]
Nvidia. Self-driving safety report, 2021. Available at https://images.nvidia.com/content/self-driving-cars/safety-report/auto-print-self-driving-safety-report-2021-update.pdf; 27 pgs. [cited by applicant]
Zhou, et al.; “Tracking Objects as Points”; arXiv:2004.01177; 22 pgs. [cited by applicant]
Weng, et al.; “MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation”; arXiv:2110.09481; Oct. 18, 2021, 7 pgs. [cited by applicant]
Weng, et al.; “Whose Track is it anyway? Improving Robustness to Tracking Errors with Affinity-based Trajectory Prediction”; In Conference on Computer Vision and Pattern Recognition, 2022, 10 pgs. [cited by applicant]
Zeng, et al.; “End-to-End Interpretable Neural Motion Planner”; In Conference on Computer Vision and Pattern Recognition, pp. 8660-8669, 2019. [cited by applicant]
Casas, et al.; “MP3: A Unified Model to Map, Perceive, Predict and Plan”; In Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 10 pgs. [cited by applicant]
Karkus, et al.; “Differentiable Algorithm Networks for Composable Robot Learning”; arXiv:1908.11602; May 28, 2019, 12 pgs. [cited by applicant]
Amos, et al.; “Differentiable MPC for End-to-End Planning and Control”; In Advances in Neural Information Processing Systems, pp. 8299-8310, 2018. [cited by applicant]
Ivanovic, et al.; “Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty”; arXiv:2104.12446; Mar. 3, 2022, 15 pgs. [cited by applicant]
Weng, et al.; “PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling”; arXiv:2003.07847; Apr. 3, 2021, 8 pgs. [cited by applicant]
Ivanovic, et al. “Propagating State Uncertainty through Trajectory Forecasting”; arXiv:2110.03267; Jul. 12, 2022, 8 pgs. [cited by applicant]
Liu, et al.; “Path Planning for Autonomous Vehicles Using Model Predictive Control”; In IEEE Intelligent Vehicles Symposium, Jun. 11-14, 2017; 6 pgs. [cited by applicant]
Ivanovic, et al.; “MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control”; arXiv:2009.07517; Jan. 14, 2021, 14 pgs. [cited by applicant]
Chen, et al.; “Reactive Motion Planning with Probabilistic Safety Guarantees”; In Conference on Robot Learning, 2020, 13 pgs. [cited by applicant]
Schaefer, et al.; “Leveraging Neural Network Gradients within Trajectory Optimization for Proactive Human-Robot Interactions”; arXiv:2012.01027; Dec. 2, 2020; 7 pgs. [cited by applicant]
Casas, et al.; “The Importance of Prior Knowledge in Precise Multimodal Prediction”; arXiv:2006.02636; Jun. 4, 2020; 10 pgs. [cited by applicant]
Ivanovic, et al.; “Injecting Planning-Awareness into Prediction and Detection Evaluation”; arXiv:2110.03270; Oct. 7, 2021; 8 pgs. [cited by applicant]
McAllister, et al.; “Control-Aware Prediction Objectives for Autonomous Driving”; arXiv:2204.13319; Apr. 28, 2022, 8 pgs. [cited by applicant]
El Sallab, et al.; “Deep Reinforcement Learning Framework for Autonomous Driving”; arXiv:1704.02532; Apr. 8, 2017, 7 pgs. [cited by applicant]
Jain, et al.; “Autonony 2.0: Why is Self-Driving Always 5 Years Away?”; arXiv:2107.08142; Aug. 9, 2021, 10 pgs. [cited by applicant]
Zeng, et al.; “DSDNet: Deep Structured Self-Driving Network”; arXiv:2008.06041; Aug. 13, 2020, 24 pgs. [cited by applicant]
Cui, et al.; “LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving”; In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16107-16116, 2021; 10 pgs. [cited by applicant]
Jonschkowski, et al.; “Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors”; arXiv:1805.11122, May 30, 2018, 9 pgs. [cited by applicant]
Karkus, et al.; “Particle Filter Networks with Application to Visual Localization”; In Proceedings of the Conference on Robot Learning, pp. 169-178, 2018. [cited by applicant]
Karkus, et al.; “Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation”; In IEEE Conference on Computer Vision and Pattern Recognition, 2021, 11 pgs. [cited by applicant]
Karkus, et al.; “QMDP-net: Deep Learning for Planning Under Partial Observability”; In Advances in Neural Information Processing Systems, pp. 4697-4707, 2017, 11 pgs. [cited by applicant]
Gupta, et al.; “Cognitive Mapping and Planning for Visual Navigation”; arXiv:1702.03920; Feb. 7, 2019, 19 pgs. [cited by applicant]
Okada, et al.; “Path Integral Networks: End-to-End Differentiable Optimal Control”; arXiv:1706.09597; Jun. 29, 2017, 12 pgs. [cited by applicant]
Pereira, et al.; “MPC-Inspired Neural Network Policies for Sequential Decision Making”; arXiv:1802.05803; Mar. 14, 2018, 10 pgs. [cited by applicant]
East, et al.; “Infinite-Horizon Differentiable Model Predictive Control”; arXiv:2001.02244; Jan. 7, 2020, 15 pgs. [cited by applicant]
Tassa, et al.; “Control-Limited Differential Dynamic Programming”; In IEEE International Conference on Robotics and Automation (ICRA), pp. 1168-1175, IEEE, 2014. [cited by applicant]
LaValle, et al.; “Better Unicycle Models”; In Planning Algorithms, pp. 743-743. Cambridge University Press, 2006, 1 pg. [cited by applicant]
Bansal, et al.; “ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst”; arXiv.1812.03079; Dec. 7, 2018, 20 pgs. [cited by applicant]
Chen, et al.; “Interactive Multi-Modal Motion Planning with Branch Model Predictive Control”; arXiv:2109.05128; Sep. 18, 2021, 10 pgs. [cited by applicant]
Chen, et al.; “ScePT: Scene-Consistent, Policy-Based Trajectory Predictions for Planning”; In Conference on Computer Vision and Pattern Recognition, 2022, 10 pgs. [cited by applicant]
Rudenko, et al.; “Human Motion Trajectory Prediction: A Survey”; arXiv:1905.06113; Dec. 17, 2019, 37 pgs. [cited by applicant]
Paszke, et al.; “Pytorch: An Imperative Style, High-Performance Deep Learning Library”; In Advances in Neural Information Processing Systems, pp. 8024-8035, 2019. [cited by applicant]
Bergamini, et al.; “SimNet: Learning Reactive Self-Driving Simulations from Real-World Observations”; arXiv:2105.12332; May 26, 2021, 7 pgs. [cited by applicant]
Suo, et al.; “TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors”; In Conference on Computer Vision and Pattern Recognition (SVPR), pp. 10400-10409, 2021. [cited by applicant]
Caesar, et al.; “NuPlan: A Closed-Loop ML-Based Planning Benchmark for Autonomous Vehicles”; arXiv:2106.11810; Feb. 4, 2022; 5 pgs. [cited by applicant]
Rempe, et al.; “Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior”; In Conference on Computer Vision and Pattern Recognition (CVPR), 2022; 11 pgs. [cited by applicant]
Weber, et al.; “Credit Assignment Techniques in Stochastic Computation Graphs”; arXiv:1901.01761; Jan. 7, 2019, 40 pgs. [cited by applicant]
Wang, et al.; “DETR3D: 3D Object Detection from Multi-View Images via 3D-to-2D Queries,” https://arxiv.org/abs/2110.06922, Oct. 13, 2021, 12 pgs. [cited by applicant]
Karkus, et al.: “Diffstack: A differentiable and modular control stack for autonomous vehicles, ”https://arxiv.org/abs/2212.06437; Dec. 13, 2022, 17 pgs. [cited by applicant]
Li, et al.; “End-to-End 3D Tracking with Decoupled Queries,” In International Conference on Computer Vision (ICCV), 2023, 10 pgs. [cited by applicant]
Xu, et al.; “How To Train Your Deep Multi-Object Tracker,” https://arxiv.org/abs/1906.06618, Apr. 23, 2020, 14 pgs. [cited by applicant]
Sahoo, et al.; “Backpropagation through Combinatorial Algorithms: Identity with Projection Works,” https://arxiv.org/ abs/2205.15213, Mar. 17, 2023, 27 pgs. [cited by applicant]
Yin, et al.; “Center-Based 3D Object Detection and Tracking,” https://arxiv.org/abs/2006.11275; Jan. 6, 2021, 12 pgs. [cited by applicant]
Lang, et al.; “PointPillars: Fast Encoders for Object Detection from Point Clouds,” https://arxiv.org/abs/1812.05784; May 7, 2019, 9 pgs. [cited by applicant]
Chen, et al.; “FUTR3D: A Unified Sensor Fusion Framework for 3D Detection,” https://arxiv.org/abs/2203.10642; Apr. 15, 2023, 13 pgs. [cited by applicant]