IP Library › Granted Patent US 12,485,890
Granted Patent B2
US 12,485,890 · App. 18/183,566 · Granted Dec 2, 2025

Learning autonomous vehicle safety concepts from demonstrations

Inventors: Karen Yan Ming Leung (Los Altos, CA); Sushant Veer (Sunnyvale, CA); Edward Fu Schmerling (Los Altos, CA); Marco Pavone (Stanford, CA)
Assignee: NVIDIA Corporation
B60W30/0956B60W30/09B60W60/0015B60W2554/4041
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,485,890
App. No.
18/183,566
Granted
Dec 2, 2025
Kind
B2
Abstract

In various examples, control policies for controlling agents may be learned from demonstrations capturing joint states of entities navigating through the environment. A control policy may be learned mapping joint states to control actions, where the joint states are between agents, and the control actions are of at least one of the agents. The control policy may be learned to define the mappings as control invariant sets of the joint sates and the control actions. The control policy may be used to determine one or more functions that compute, based at least on a joint state between entities, output indicating a likelihood of collision between the entities operating in accordance with the control policy. Using the output, current and/or potential states of the environment may be evaluated to determine control operations for a machine, such as a vehicle.

Claims (79)

1 . A method comprising:

applying, to at least one machine learning model (MLM), a dataset corresponding to sensor data obtained using one or more sensors of a plurality of machines, the dataset representing examples of collision-free trajectories using joint states of entities navigating through an environment;

based at least on the applying, training the at least one MLM to learn a control policy defining a control action space for a machine of the plurality of the machines and to predict, using the control policy and for a joint state between agents, one or more control actions of the control action space for at least one agent of the agents to perform in response to the joint state, the control policy being learned using a function that assigns values to states resulting from control actions of the control action space, the values defining whether the states are within an unsafe region of a state space or a safe region of the state space based at least on a threshold value that defines a boundary between the safe region and the unsafe region, the safe region corresponding to the collision-free trajectories;

determining one or more functions that compute, based at least on the joint state and for the one or more control actions, output indicating a likelihood of collision if the at least one agent were to use the one or more control actions predicted by the at least one MLM using the control policy; and

performing one or more control operations for the machine using the output indicating the likelihood of collision.

2 . The method of claim 1 , wherein the control policy learned by the at least one MLM predicts control-invariant sets comprising joint states and corresponding control actions, the corresponding control actions, when used, resulting in future joint states that remain with the control action space corresponding to the collision-free trajectories.

3 . The method of claim 1 , wherein the joint state includes one or more of:

one or more positions of the agents;

one or more velocities of the agents; or

one or more headings of the agents.

4 . The method of claim 1 , wherein a control action of the one or more control actions includes one or more of:

one or more steering parameters; or

one or more acceleration parameters.

5 . The method of claim 1 , wherein the function includes a control barrier function.

6 . The method of claim 1 , wherein the one or more control operations correspond to a control action and the control action is determined based at least on:

determining, using the one or more functions, a set of control actions; and

selecting the control action from the set of control actions.

7 . The method of claim 1 , wherein the likelihood of collision corresponds to one or more distances of one or more first joint states from one or more second joint states corresponding to the collision between the entities if the at least one agent were to use the one or more control actions predicted by the one or more MLMs.

8 . The method of claim 1 , wherein the one or more functions are separate from the at least one MLM, and one or more parameters of the one or more functions are learned jointly with the at least one MLM learning the control policy.

9 . The method of claim 1 , wherein the entities include the machine.

10 . A system comprising:

one or more circuits to perform operations including:

applying, to at least one machine learning model (MLM), a dataset corresponding to sensor data obtained using one or more sensors of a plurality of machines, the dataset representing examples of collision-free trajectories using joint states of entities navigating through an environment;

based at least on the applying, training the at least one MLM to learn a control policy defining a control action space for a machine of the plurality of machines and to predict, using the control policy and for a joint state between agents, one or more control actions of the control action space for at least one agent of the agents to perform in response to the joint state, the control policy being learned using a function that assigns values to states resulting from control actions of the control action space, the values defining whether the states are within an unsafe region of a state space or a safe region of the state space based at least on a threshold value that defines a boundary between the safe region and the unsafe region, the safe region corresponding to the collision-free trajectories;

determining one or more functions that compute, based at least on the joint state and for the one or more control actions, output indicating a likelihood of collision if the at least one agent were to use the one or more control actions predicted by the at least one MLM using the control policy; and

performing one or more control operations for the machine using the output indicating the likelihood of collision.

11 . The system of claim 10 , further comprising determining the joint state based at least on analyzing sensor data obtained using one or more sensors of the machine.

12 . The system of claim 10 , wherein the function includes a higher order control barrier function.

13 . The system of claim 10 , wherein one or more functions are separate from the at least one MLM, and one or more parameters of the one or more functions are learned jointly with the at least one MLM learning the control policy.

14 . The system of claim 10 , wherein the joint state includes one or more of:

one or more positions of the agents;

one or more velocities of the agents; or

one or more headings of the agents.

15 . The system of claim 10 , wherein the system is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

16 . At least one processor comprising:

one or more circuits of a machine to:

determine, using sensor data obtained using one or more sensors, a joint state comprising positions of the machine and an agent in an environment;

apply the positions of the machine and the agent to a neural network that is trained, using a dataset representing demonstrations of collision-free trajectories using joint states of entities navigating through the environment, to learn a control policy defining a control action space for the machine and to predict, using the control policy and for the joint state, one or more control actions of the control action space for the machine, the control policy being learned using a function that assigns values to states resulting from control actions of the control action space, the values defining whether the states are within an unsafe region of a state space or a safe region of the state space based at least on a threshold value that defines a boundary between the safe region and the unsafe region, the safe region corresponding to the collision-free trajectories;

compute, using one or more functions, a likelihood of collision between the machine and the agent, were the machine to use the one or more control actions that were predicted by the neural network; and

perform one or more control operations for the machine based on the likelihood of collision.

17 . The at least one processor claim 16 , wherein the neural network outputs a representation of the one or more control actions and the one or more functions output the likelihood of collision.

18 . The at least one processor claim 17 , wherein the joint state includes one or more of:

one or more positions of the agent and the machine;

one or more velocities of the agent and the machine; or

one or more headings of the agent or the machine.

19 . The at least one processor claim 16 , wherein a control action of the one or more control actions includes one or more of:

one or more steering parameters; or

one or more acceleration parameters.

20 . The at least one processor claim 16 , wherein the at least one processor is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 6, 2023
From: LEUNG, KAREN YAN MING; VEER, SUSHANT; SCHMERLING, EDWARD FU; PAVONE, MARCO
To: NVIDIA CORPORATION
Reel/Frame 064167/0323 →
Continuity (2)
Provisional Application 63359414 · Jul 8, 2022
Related Publication 20240010196A1 · Jan 11, 2024
References Cited (64)
US 11807233B1 · Clawson · 2023 [cited by examiner]
US 20180141544A1 · Xiao · 2018 [cited by examiner]
US 20190258251A1 · Ditty et al. · 2019 [cited by applicant]
US 20190265703A1 · Hicok et al. · 2019 [cited by applicant]
US 20190384303A1 · Muller et al. · 2019 [cited by applicant]
US 20210253128A1 · Nister et al. · 2021 [cited by applicant]
US 20210349460A1 · Huang et al. · 2021 [cited by applicant]
US 20220388540A1 · Fu · 2022 [cited by examiner]
Van Koevering, et al.; “Provable Probabilistic Safety and Feasibility-Assured Control for Autonomous Vehicles using Exponential Control Barrier Functions,” https://arxiv.org/abs/2205.03744; May 8, 2022, 6 pgs. [cited by applicant]
Taylor, et al.; “Adaptive Safety with Control Barrier Functions,” https://arxiv.org/abs/1910.00555; Oct. 1, 2019, 8 pgs. [cited by applicant]
Xiao, et al.; “Adaptive Control Barrier Functions,” IEEE Transactions on Automatic Control, 2021. [cited by applicant]
Xiao, et al.; “Control Barrier Functions for Systems with Multiple Control Inputs,” https://arxiv.org/abs/2203.07978; Mar. 15, 2022, 6 pgs. [cited by applicant]
Tonkens, et al.; “Refining Control Barrier Functions Through Hamilton-Jacobi Reachability,” https://arxiv.org/abs/2204.12507; Aug. 19, 2022, 8 pgs. [cited by applicant]
Choi, et al.; “Robust Control Barrier-Value Functions for Safety-Critical Control,” https://arxiv.org/abs/2104.02808; Oct. 25, 2021, 9 pgs. [cited by applicant]
Mitchell, I.M.; et al.; “A Time-Dependent Hamilton-Jacobi Formulation of Reachable Sets for Continuous Dynamic Games”, IEEE Transactions on Automatic Control, vol. 50, No. 7, pp. 947-957, 2005. [cited by applicant]
Margellos, Kostas, et al.; “Hamilton-Jacobi Formulation for Reach-Avoid Differential Games”, IEEE Transaction on Automatic Control, vol. 56, No. 8, pp. 1849-1861, 2011. [cited by applicant]
Leung, K., et al.; “Towards the Unification and Data-Driven Synthesis of Autonomous Vehicle Safety Concepts”; https://arxiv.org/abs/2107.14412, Jun. 20, 2022, 9 pgs. [cited by applicant]
Shalev-Shwartz, S., et al.; “On a Formal Model of Safe and Scalable Self-Driving Cars”; https://arxiv.org/abs/1708.06374, Oct. 27, 2018, 27 pgs. [cited by applicant]
Leung, K., et al.; “On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions”; https://arxiv.org/abs/2012.03390, Dec. 6, 2020, 18 pgs. [cited by applicant]
Bradbury, J., et al.; “JAX: composable transformations of Python+NumPy programs”, 2018. URL http://github.com/google/jax. [cited by applicant]
Nister, D. et al., “An Introduction to the Safety Force Field”, NVIDIA, Mar. 2019, 27 pgs. [cited by applicant]
“IEEE Standard for Assumptions in Safety-Related Models for Automated Driving Systems,” IEEE Std 2846-2022, 2022. [cited by applicant]
Van den Berg, et al.; “Reciprocal Velocity Obstacles for Real-Time Multi-Agent Navigation,” in IEEE International Conference on Robotics and Automation, 2008. [cited by applicant]
Chen, et al.; “Multi-Vehicle Collision Avoidance via Hamilton-Jacobi Reachability and Mixed Integer Programming,” in IEEE Conference on Decision and Control, 2016. [cited by applicant]
Wang, et al.; “Infusing Reachability-Based Safety into Planning and Control for Multi-Agent Interactions,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020. [cited by applicant]
Guo, et al.; “VR-ORCA: Variable Responsibility Optimal Reciprocal Collision Avoidance,” IEEE Robotics and Automation Letters, vol. 6, No. 3, pp. 4520-4527, 2021. [cited by applicant]
Robey, et al.; “Learning Control Barrier Functions from Expert Demonstrations,” in IEEE Conference on Decision and Control, 2020, pp. 3717-3724. [cited by applicant]
Qin, et al.; “Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates,” International Conference on Learning Representations, 2021. [cited by applicant]
Ziebart, et al.; “Maximum Entropy Inverse Reinforcement Learning,” in AAAI Conference on Artificial Intelligence, vol. 8, 2008, pp. 1433-1438. [cited by applicant]
Wilkie, et al.; “Generalized Velocity Obstacles,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009, pp. 5573-5578. [cited by applicant]
Farid, et al.; “Task-Driven Out-of-Distribution Detection with Statistical Guarantees for Robot Learning”; Proceedings of the Conference on Robot Learning, 2021, 11 pgs. [cited by applicant]
Althoff, et al.; “Online Verification of Automated Road Vehicles Using Reachability Analysis”; 30(4):903-918, 2014, 16 pgs. [cited by applicant]
Tian, et al.; “Safety Assurances for Human-Robot Interaction via Confidence-Aware Game-Theoretic Human Models”; arXiv:2109.14700; Oct. 30, 2021, 7 pgs. [cited by applicant]
Farid, et al.; “Failure Prediction with Statistical Guarantees for Vision-Based Robot Control”; arXiv:2202.05894; May 6, 2022, 13 pgs. [cited by applicant]
Rudenko, et al.; “Human Motion Trajectory Predication: A Survey,” https://arxiv.org/abs/1905.06113; Dec. 17, 2019; 37 pgs. [cited by applicant]
Phan-Minh, et al.; “Driving in Real Life with Inverse Reinforcement Learning,” https://arxiv.org/abs/2206.03004; Jun. 7, 2022, 23 pgs. [cited by applicant]
Oboril, et al.; “MTBF Model for AVs—from perception errors to vehicle-level failures,” 2022. [cited by applicant]
Abbeel, et al.; “Apprenticeship Learning via Inverse Reinforcement Learning,” In Int. Conf. on Machine Learning, 2004, 8 pgs. [cited by applicant]
Levine, et al.; “Continuous Inverse Optimal Control with Locally Optimal Examples,” https://arxiv.org/abs/1206.4617; 2012, 8 pgs. [cited by applicant]
Sadigh, et al., “Planning for Autonomous Cars That Leverage Effects on Human Actions,” In Robotics: Science and Systems, 2016, 9 pgs. [cited by applicant]
Wu, et al.; “Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving,” https://arxiv.org/abs/2006.13704; Jun. 22, 2020, 9 pgs. [cited by applicant]
Xiao, et al., “High Order Control Barrier Functions,” IEEE Transactions on Automatic Control, 2021. [cited by applicant]
Fiorini, et al.; “Motion Planning in Dynamic Environments using Velocity Obstacles,” Int. Journal of Robotics Research, 17(10-11):760-772, 1998, 27 pgs. [cited by applicant]
Holmes, et al.; “Reachable sets for safe, real-time manipulator trajectory design,” https://arxiv.org/abs/2002.01591, Sep. 29, 2020, 14 pgs. [cited by applicant]
Kuwata, et al.; “Real-time Motion Planning with Applications to Autonomous Urban Driving,” IEEE Transactions on Control Systems Technology, vol. 17, No. 5, Sep. 2009, 14 pgs. [cited by applicant]
Janson, et al.; “Safe Motion Planning in Unknown Environments: Optimally Benchmarks and Tractable Policies,” https://arxiv.org/abs/1804.05804; Apr. 16, 2018, 10 pgs. [cited by applicant]
Fridovich-Keil, et al.; “Confidence-Aware Motion Prediction for Real-time Collision Avoidance,” In Int. Journal of Robotics Research, 2019. [cited by applicant]
Ames, et al.; “Control Barrier Functions: Theory and Applications,” https://arxiv.org/abs/1903.11199, Mar. 27, 2019, 12 pgs. [cited by applicant]
Robey, et al.; “Learning Robust Hybrid Control Barrier Functions for Uncertain Systems,” https://arxiv.org/abs/2101.06492, May 13, 2021, 17 pgs. [cited by applicant]
Prabhakar, et al.; “Credit Assignment Safety Learning from Human Demonstrations,” https://arxiv.org/abs/2110.04633; Oct. 9, 2021, 5 pgs. [cited by applicant]
Lyu, et al.; “Adaptive Safe Merging Control for Heterogeneous Autonomous Vehicles using Parametric Control Barrier Functions,” https://arxiv.org/abs/2202.09936; Feb. 20, 2022, 6 pgs. [cited by applicant]
Clark, “Verification and Synthesis of Control Barrier Functions”, https://arxiv.org/abs/2104.14001; Apr. 28, 2021, 8 pgs. [cited by applicant]
Sill; “Monotonic Networks,” In Conf. on Neural Information Processing Systems, 1997, 7 pgs. [cited by applicant]
Wehenkel, et al.; “Unconstrained Monotonic Neural Networks,” https://arxiv.org/abs/1908.05164; Mar. 31, 2021, 14 pgs. [cited by applicant]
Chen, et al.; “Counter-Example Guided Learning of Bounds on Environmental Behavior,” https://arxiv.org/abs/2001.07233; Feb. 6, 2020, 11 pgs. [cited by applicant]
Von Neumann; “Zur Theorie der Gesellschaftsspiele” Mathematische Annalen, 100(1):295-320, 1927, 26 pgs. [cited by applicant]
Schmerling, et al.; “Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction,” https://arxiv.org/abs/1710.09483; Oct. 25, 2017, 8 pgs. [cited by applicant]
“Next Generation Simulation (NGSIM) Vehicle Trajectories and Supporting Data,” U.S. Department of Transportation Intelligent Transportation Systems Joint Program, 2016. [cited by applicant]
Chen, et al.; “Exact and Efficient Hamilton-Jacobi-based Guaranteed Safety Analysis via System Decomposition,” https://arxiv.org/abs/1609.05248; Sep. 16, 2016; 9 pgs. [cited by applicant]
Herbert, et al.; “Scalable learning of safety guarantees for autonomous systems using Hamilton-Jacobi reachability,” https://arxiv.org/abs/2101.05916; Apr. 2, 2021, 7 pgs. [cited by applicant]
Bansal, et al.; “DeepReach: A Deep Learning Approach to High-Dimensional Reachability,” https://arxiv.org/abs/2011.02082; Nov. 4, 2020, 9 pgs. [cited by applicant]
Fisac, et al.; “Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning,” In Proc. IEEE Conf. on Robotics and Automation, 2019. [cited by applicant]
Tsaknakis, et al.; “Minimax Problems with Coupled Linear Constraints: Computational Complexity, Duality and Solution Methods,” https://arxiv.org/abs/2110.11210; Nov. 26, 2022, 29 pgs. [cited by applicant]
Goktas, et al.; “Convex-Concave Min-Max Stackelberg Games,” https://arxiv.org/abs/2110.05192' Jul. 5, 2023, 34 pgs. [cited by applicant]