IP Library › Granted Patent US 12,441,366
Granted Patent B2
US 12,441,366 · App. 18/183,582 · Granted Oct 14, 2025

Task-relevant failure detection for trajectory prediction in machines

Inventors: Alec Farid (Princeton, NJ); Sushant Veer (Sunnyvale, CA); Boris Ivanovic (Mountain View, CA); Karen Yan Ming Leung (Los Altos, CA); Marco Pavone (Stanford, CA)
Assignee: NVIDIA Corporation
B60W60/0016B60W30/09B60W60/00272
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,441,366
App. No.
18/183,582
Granted
Oct 14, 2025
Kind
B2
Abstract

In various examples, cost probability distributions corresponding to predicted locations of an object in an environment and potential locations for a machine in the environment and may be evaluated using corresponding observed costs corresponding to the machine and the object. The cost probability distributions may be evaluated based on comparing the observed costs to threshold values, which may be determined based on sampling a predicted cost function. A threshold value may be selected to provide false-positive rate and/or false-negative rate guarantees for anomaly detection. Control operations may be performed based on results of the evaluation of the cost probability distributions. For example, based on the results, a motion planner may reuse a planned trajectory for a future planning cycle (e.g., thereby avoiding re-planning computations) or generate and/or select a new planned trajectory (e.g., based at least on one or more anomalies being detected).

Claims (89)

1. A system comprising:

one or more processing units to perform operations including:

computing, using sensor data generated using one or more sensors, a predicted trajectory of an actor in an environment and one or more observed locations of the actor in the environment;

determining based at least on a planned trajectory for a vehicle in the environment, a cost probability distribution indicating cost values across different potential locations for the actor relative to the predicted trajectory, the different potential locations corresponding to a future position distribution associated with the predicted trajectory for the actor;

performing anomaly detection based at least on comparing an observed cost corresponding to the one or more observed locations of the actor in the environment to the cost probability distribution; and

performing one or more control operations for the vehicle based at least on one or more results of the anomaly detection.

2. The system of claim 1 , wherein the performing the anomaly detection includes:

generating samples of the cost probability distribution;

determining a threshold value using the samples of the cost probability distribution; and

comparing the observed cost to the threshold value to determine the one or more results.

3. The system of claim 1 , wherein the cost probability distribution is based at least on one or more of:

one or more spatial distances between the vehicle and the actor;

one or more times-to-collision between the vehicle and the actor;

one or more speed limits associated with the vehicle;

one or more comfort parameters associated with the vehicle; or

one or more lane heading parameters associated with the vehicle.

4. The system of claim 1 , further comprising:

determining a threshold value based at least on one or more of a false-positive rate or a false-negative rate specified by a user; and

determining the one or more results based at least on comparing the observed cost to the threshold value.

5. The system of claim 1 , wherein the planned trajectory includes a planned trajectory for the vehicle and the one or more control operations include replacing the planned trajectory for the vehicle based at least on the one or more results.

6. The system of claim 1 , wherein the planned trajectory is determined using a first motion planning cycle and the one or more control operations include determining, using a second motion planning cycle to reuse the planned trajectory for the vehicle based at least on the one or more results.

7. The system of claim 1 , wherein the cost probability distribution indicates the cost values across different potential locations for the actor at a single time step of the predicted trajectory, the different potential locations correspond to samples of the future position distribution at the single time step, and the one or more observed locations correspond to an actual location of the actor at the single time step.

8. The system of claim 1 , wherein the one or more results include an identifier of the actor in association with one or more detected anomalies, and the one or more control operations are based at least on the one or more results including the identifier of the actor in association with the one or more detected anomalies.

9. The system of claim 1 , wherein the system is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3 D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

10. A method comprising:

determining, based at least on one or more potential locations for a machine in an environment, a cost probability distribution indicating cost values across different potential locations for an object in an environment relative to one or more predicted locations for the object, the different potential locations corresponding to a future position distribution associated with the one or more predicted locations for the object;

computing, using one or more observed locations of the object in the environment, an observed cost corresponding to the machine and the object; and

performing one or more control operations for the machine based at least on one or more results of comparing the observed cost to the cost probability distribution.

11. The method of claim 10 , wherein the comparing indicates whether the observed cost has a threshold level of similarity to the cost values indicated by the cost probability distribution.

12. The method of claim 10 , comprising:

generating samples of the cost probability distribution; and

determining a threshold value using the samples of the cost probability distribution;

wherein the comparing includes determining whether the observed cost exceeds the threshold value to determine the one or more results.

13. The method of claim 10 , wherein the cost probability distribution is based at least on one or more of:

one or more spatial distances between the machine and the object;

one or more times-to-collision between the machine and the object;

one or more speed limits;

one or more comfort parameters; or

one or more lane heading parameters.

14. The method of claim 10 , further comprising:

determining a threshold value based at least on one or more of a false-positive rate or a false-negative rate specified by a user; and

determining the one or more results based at least on comparing the observed cost to the threshold value.

15. The method of claim 10 , wherein the one or more potential locations correspond to a planned trajectory for the machine and the one or more control operations include replacing the planned trajectory for the machine based at least on the one or more results.

16. At least one processor comprising:

one or more circuits to perform one or more control operations for a machine based at least on one or more results of comparing an observed cost to a cost probability distribution, the comparing being based at least on:

determining, based at least on one or more potential locations for the machine in an environment, the cost probability distribution indicating cost values across different potential locations for an object in an environment relative to one or more predicted locations for the object, the different potential locations corresponding to a future position distribution associated with the one or more predicted locations for the object; and

computing, using one or more observed locations of the object in the environment, the observed cost corresponding to the machine and the object.

17. The at least one processor of claim 16 , wherein the comparing includes:

generating samples of the cost probability distribution;

determining a threshold value using the samples of the cost probability distribution; and

comparing the observed cost to the threshold value.

18. The at least one processor of claim 16 , wherein the cost probability distribution is based at least on one or more of:

one or more spatial distances between the machine and the object;

one or more times-to-collision between the machine and the object;

one or more speed limits associated with the machine;

one or more comfort parameters associated with the machine; or

one or more lane heading parameters associated with the machine.

19. The at least one processor of claim 16 , wherein the one or more circuits are to determine a threshold value based at least on one or more of a false-positive rate or a false-negative rate specified by a user, and the comparing includes comparing the observed cost to the threshold value.

20. The at least one processor of claim 16 , wherein the at least one processor is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 17, 2023
From: FARID, ALEC; VEER, SUSHANT; IVANOVIC, BORIS; LEUNG, KAREN YAN MING; PAVONE, MARCO
To: NVIDIA CORPORATION
Reel/Frame 063014/0955 →
Continuity (2)
Provisional Application 63359416 · Jul 8, 2022
Related Publication 20240017743A1 · Jan 18, 2024
References Cited (57)
US 11126180B1 · Kobilarov · 2021 [cited by examiner]
US 11810365B1 · Crego · 2023 [cited by examiner]
US 20190266418A1 · Xu et al. · 2019 [cited by applicant]
US 20190384303A1 · Muller et al. · 2019 [cited by applicant]
US 20190384304A1 · Towal · 2019 [cited by examiner]
US 20200249684A1 · Onofrio et al. · 2020 [cited by applicant]
US 20200341466A1 · Pham et al. · 2020 [cited by applicant]
US 20200377085A1 · Floyd-Jones · 2020 [cited by examiner]
US 20210046924A1 · Caldwell · 2021 [cited by examiner]
US 20230227061A1 · Ziebart · 2023 [cited by examiner]
Leung, K., et al.; “On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions”; https://arxiv.org/abs/2012.03390, Dec. 6, 2020, 18 pgs. [cited by applicant]
Liu, et al.; “Open Category Detection with PAC Guarantees”; Proceedings of the International Conference on Machine Learning, 2018, 10 pgs. [cited by applicant]
Farid, et al.; “Task-Driven Out-of-Distribution Detection with Statistical Guarantees for Robot Learning”; Proceedings of the Conference on Robot Learning, 2021, 11 pgs. [cited by applicant]
Greenberg, et al.; “Detecting Rewards Deterioration in Episodic Reinforcement Learning”; In International Conference on Machine Larning, V. 139, pp. 3842-3853, 2021. [cited by applicant]
Caesar, H., et al.: “NuScenes: A Multimodal Dataset for Autonomous Driving”; https://arxiv.org/abs/1903.11027; May 5, 2020, 16 pgs. [cited by applicant]
Caesar, et al.; “NuPlan: A Closed-Loop ML-Based Planning Benchmark for Autonomous Vehicles”; arXiv:2106.11810; Feb. 4, 2022; 5 pgs. [cited by applicant]
“IEEE Standard for Assumptions in Safety-Related Models for Automated Driving Systems,” IEEE Std 2846-2022, 2022. [cited by applicant]
Nister, D. et al., “An Introduction to the Safety Force Field”, NVIDIA, Mar. 2019, 27 pgs. [cited by applicant]
Shalev-Shwartz, S., et al.; “On a Formal Model of Safe and Scalable Self-Driving Cars”; https://arxiv.org/abs/1708.06374, Oct. 27, 2018, 27 pgs. [cited by applicant]
Althoff, et al.; “Online Verification of Automated Road Vehicles Using Reachability Analysis”; 30(4):903-918, 2014, 16 pgs. [cited by applicant]
General Motors. Self-Driving Safety Report, 2018. Available at https://www.gm.com/content/dam/company/docs/US/en/gmcom/gmsafetyreport.pdf.; 33 pgs. [cited by applicant]
Argo AI. Developing a self-driving system you can trust, Apr. 2021. Available at https://www.argo.ai/wp-content/uploads/2021/04/AargoSafetyReport.pdf.; 55 pgs. [cited by applicant]
Zoox. Safety report vol. 2.0, 2021. Available at https://zoox.com/safety/. [cited by applicant]
Waymo. Safety report, 2021.Available at https://waymo.com/safety/safety-report. Retrieved on Jul. 4, 2021. [cited by applicant]
NVIDIA. Self-driving safety report, 2021. Available at https://images.nvidia.com/content/self-driving-cars/safety-report/auto-print-self-driving-safety-report-2021-update.pdf; 27 pgs. [cited by applicant]
Dahl, J.; “Collision Avoidance: A Literature Review on Threat-Assessment Techniques”; IEEE Transactions on Intelligent Vehicles, vol. 4, No. 1, pp. 101-113, 2019. [cited by applicant]
Mitchell, I.M.; et al.; “A Time-Dependent Hamilton-Jacobi Formulation of Reachable Sets for Continuous Dynamic Games”, IEEE Transactions on Automatic Control, vol. 50, No. 7, pp. 947-957, 2005. [cited by applicant]
Hsu, et al.; “Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees”; arXiv:2201.08355; Apr. 1, 2023; 28 pgs. [cited by applicant]
Hu, et al.; “SHARP: Shielding-Aware Robust Planning for Safe and Efficient Human-Robot Interaction”; arXiv:2110.00843, Mar. 10, 2022, 8 pgs. [cited by applicant]
Leung, et al.; “Toward the Unification and Data-Driven Synthesis of Autonomous Vehicle Safety Concepts”; arXiv:2107.14412; Jun. 20, 2022, 9 pgs. [cited by applicant]
Tian, et al.; “Safety Assurances for Human-Robot Interaction via Confidence-Aware Game-Theoretic Human Models”; arXiv:2109.14700; Oct. 30, 2021, 7 pgs. [cited by applicant]
Basseville; “Detecting Changes in Signals and Systems—A Survey”; In Automatica, V.24, pp. 309-326, 1988; 47 pgs. [cited by applicant]
Hendrycks, et al.; “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”; arXiv:1610.02136; Oct. 3, 2018, 12 pgs. [cited by applicant]
Lee, et al.; “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks”; In Advances in Neural Information Processing Systems, 2018, 11 pgs. [cited by applicant]
Hendrycks, et al.; “Scaling Out-of-Distribution Detection for Real-World Settings”; arXiv:1911.11132; May 15, 2022, 14 pgs. [cited by applicant]
Ruff, et al.; “A Unifying Review of Deep and Shallow Anomaly Detection”; In Proceedings of the IEEE, V.109, pp. 756-795, 2021, 40 pgs. [cited by applicant]
Sedlmeier, et al.; “Uncertainty-Based Out-of-Distribution Classification in Deep Reinforcement Learning”; arXiv:2001.00496, Dec. 31, 2019, 10 pgs. [cited by applicant]
Sedlmeier, et al.; “Policy Entropy for Out-of-Distribution Classification”; arXiv:2005.12069, May 25, 2020, 12 pgs. [cited by applicant]
Cai, et al.; “Real-Time Out-of-Distribution Detection in Learning-Enabled Cyber-Physical Systems”; arXiv:2001.10494; Jan. 28, 2020, 10 pgs. [cited by applicant]
Wu, et al.; “RLAD: Time Series Anomaly Detection Through Reinforcement Learning and Active Learning”; arXiv:2104.00543; Mar. 31, 2021, 8 pgs. [cited by applicant]
Siddiqui, et al.; “Finite Sample Complexity of Rare Pattern Anomaly Detection”; Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2016, 10 pgs. [cited by applicant]
Farid, et al.; “Failure Prediction with Statistical Guarantees for Vision-Based Robot Control”; arXiv:2202.05894; May 6, 2022, 13 pgs. [cited by applicant]
Rudenko, et al.; “Human Motion Trajectory Prediction: A Survey”; arXiv:1905.06113; Dec. 17, 2019, 37 pgs. [cited by applicant]
Ivanovic, et al.; “Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty”; arXiv:2104.12446; Mar. 3, 2022, 15 pgs. [cited by applicant]
Weng, et al.; “PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling”; arXiv:2003.07847; Apr. 3, 2021, 8 pgs. [cited by applicant]
Weng, et al.; “MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation”; arXiv:2110.09481; Oct. 18, 2021, 7 pgs. [cited by applicant]
Liu, et al.; “Path Planning for Autonomous Vehicles Using Model Predictive Control”; In IEEE Intelligent Vehicles Symposium, Jun. 11-14, 2017; 6 pgs. [cited by applicant]
Fan, et al.; “Baidu Apollo Em Motion Planner”, arXiv:1807.08048; Jul. 20, 2018, 15 pgs. [cited by applicant]
Zeng, et al.; “End-to-End Interpretable Neural Motion Planner”; In Conference on Computer Vision and Pattern Recognition, pp. 8660-8669, 2019. [cited by applicant]
Ivanovic, et al.; “MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control”; arXiv:2009.07517; Jan. 14, 2021, 14 pgs. [cited by applicant]
Chen, et al.; “Reactive Motion Planning with Probabilistic Safety Guarantees”; In Conference on Robot Learning, 2020, 13 pgs. [cited by applicant]
Schaefer, et al.; “Leveraging Neural Network Gradients within Trajectory Optimization for Proactive Human-Robot Interactions”; arXiv:2012.01027; Dec. 2, 2020; 7 pgs. [cited by applicant]
Ivanovic, et al.; “Injecting Planning-Awareness into Prediction and Detection Evaluation”; arXiv:2110.03270; Oct. 7, 2021; 8 pgs. [cited by applicant]
McAllister, et al.; “Control-Aware Prediction Objectives for Autonomous Driving”; arXiv:2204.13319; Apr. 28, 2022, 8 pgs. [cited by applicant]
Salzmann, et al.; “Trajectron++:Dynamically-feasible Trajectory Forecasting with Heterogeneous Data”; In European Conference on Computer Vision, pp. 683-700, Springer, 2020. [cited by applicant]
Schmerling, et al.; “Multimodal Probablistic Model-Based Planning for Human-Robot Interaction”; arXiv:1710.09483; Oct. 25, 2017, 8 pgs. [cited by applicant]
Kianfar, et al.; “Safety Verification of Automated Driving Systems”; IEEE Intelligent Transportation Systems Magazine, 5(4): 73-86, 2013. [cited by applicant]