IP Library › Granted Patent US 12,566,439
Granted Patent B2
US 12,566,439 · App. 17/942,551 · Granted Mar 3, 2026

Determining perception zones for object detection in autonomous systems and applications

Inventors: Sever Ioan Topan (Burnaby, CA); Karen Yan Ming Leung (Los Altos, CA); Yuxiao Chen (Sunnyvale, CA); Pritish Tupekar (Santa Clara, CA); Edward Fu Schmerling (Los Altos, CA); Hans Jonas Nilsson (Los Gatos, CA); Michael Cox (Menlo Park, CA); Marco Pavone (Stanford, CA)
Assignee: NVIDIA Corporation
G05D1/0214G05D1/0253G06V20/58G06V2201/07
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,566,439
App. No.
17/942,551
Granted
Mar 3, 2026
Kind
B2
Abstract

In various examples, techniques for determining perception zones for object detection are described. For instance, a system may use a dynamic model associated with an ego-machine, a dynamic model associated with an object, and one or more possible interactions between the ego-machine and the object to determine a perception zone. The system may then perform one or more processes using the perception zone. For instance, if the system is validating a perception system of the ego-machine, the system may determine whether a detection error associated with the object is a safety-critical error based on whether the object is located within the perception zone. Additionally, if the system is executing within the ego-machine, the system may determine whether the object is a safety-critical object based on whether the object is located within the perception zone.

Claims (96)

1 . A method comprising:

determining one or more first parameters corresponding to an object in an environment, the one or more first parameters including at least a first direction of travel of the object;

determining one or more second parameters corresponding to an ego-machine in the environment, the one or more second parameters including at least a second direction of travel of the ego-machine;

determining a perception zone based at least on the one or more first parameters, the one or more second parameters, and an assumption that the object is going to change from navigating in the first direction of travel to navigating in a third direction of travel to attempt to collide with the ego-machine, the perception zone extending from the ego-machine and at least partially towards the object based at least on the third direction of travel; and

causing the ego-machine to navigate based at least on the perception zone.

2 . The method of claim 1 , further comprising:

determining that the object is located within the perception zone,

wherein the causing the ego-machine to navigate is performed in view of the object being located within the perception zone.

3 . The method of claim 1 , further comprising:

determining that the object includes a safety-critical object based at least on the object being located within the perception zone,

wherein the causing the ego-machine to navigate is based at least on the object including the safety-critical object.

4 . The method of claim 1 , further comprising:

determining that the object is located outside of the perception zone,

wherein the causing the ego-machine to navigate is performed in view of the object being located outside of the perception zone.

5 . The method of claim 1 , further comprising:

determining that the object does not include a safety-critical object based at least on the object being located outside of the perception zone,

wherein the causing the ego-machine to navigate is based at least on the object not including the safety-critical object.

6 . The method of claim 1 , wherein at least one of:

the one or more first parameters further include at least one of a type of the object, a location of the object, a velocity of the object, an acceleration of the object, a deceleration of the object, a size of the object, or one or more steering limits for the object; or

the one or more second parameters further include at least one of a type of the ego-machine, a location of the ego-machine, a velocity of the ego-machine, an acceleration of the ego-machine, a deceleration of the ego-machine, a size of the ego-machine, or one or more steering limits for the ego-machine.

7 . The method of claim 1 , wherein the determining the perception zone is further based at least on at least one of:

an assumption that the ego-machine navigates towards the object;

an assumption that a surface between the ego-machine and the object is flat; or

an assumption that no other object is present between the ego-machine and the object.

8 . The method of claim 1 , wherein at least one of the determining the one or more first parameters or the determining the one or more second parameters is based at least on data generated using at least one of:

a real-world sensor of the ego-machine;

a virtual sensor of a virtual representation of the ego-machine; or

a simulation engine.

9 . The method of claim 1 , wherein the determining the perception zone comprises determining, using one or more reachability algorithms, the perception zone based at least on the one or more first parameters, the one or more second parameters, and the assumption that the object is going to change from navigating in the first direction of travel to navigating in the third direction of travel to attempt to collide with the ego-machine.

10 . The method of claim 1 , wherein at least one of the determining the one or more first parameters or the determining the one or more second parameters uses one or more neural networks (NNs).

11 . A system comprising:

one or more processors to:

determine one or more first parameters associated with an object located within an environment, the one or more first parameters including at least a first location and a first direction of travel of the object;

determine one or more second parameters associated with an ego-machine located within the environment, the one or more second parameters including at least a second location and a second direction of travel of the ego-machine;

determine, based at least on the second direction of travel of the ego-machine and an assumption that the object is going to change from navigating in the first direction of travel to navigate in a third directed of travel to attempt to collide with the ego-machine, a perception zone corresponding to the ego-machine, the perception zone including an area of the environment that extends from the second location of the ego-machine and at least partially towards the first location of the object; and

cause, based at least on the perception zone and a determination of whether the object is located within the perception zone, the ego-machine to navigate.

12 . The system of claim 11 , wherein the one or more processors are further to:

determine whether the object is a safety-critical object based at least on whether the object is located in the perception zone,

wherein the ego-machine is caused to navigate based at least on the determination of whether the object is the safety-critical object.

13 . The system of claim 11 , wherein at least one of:

the one or more first parameters further include at least one of a type of the object, a velocity of the object, an acceleration of the object, a deceleration of the object, a size of the object, or one or more steering limits for the object; or

the one or more second parameters further include at least one of a type of the ego-machine, a velocity of the ego-machine, an acceleration of the ego-machine, a deceleration of the ego-machine, a size of the ego-machine, or one or more steering limits for the ego-machine.

14 . The system of claim 11 , wherein the perception zone is further determined based at least on at least one of:

an assumption that the ego-machine navigates towards the object;

an assumption that the object navigates towards the ego-machine;

an assumption that a surface between the ego-machine and the object is flat; or

an assumption that no other object is present between the ego-machine and the object.

15 . The system of claim 11 , wherein the system is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing real-time streaming;

a system for generating or presenting virtual reality (VR) content;

a system for generating or presenting augmented reality (AR) content;

a system for generating or presenting mixed reality (MR) content;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

16 . One or more processors comprising:

processing circuitry to:

determine, based at least on sensor data obtained using one or more sensors of an ego-machine, at least a first dynamic model indicating at least a first direction of travel of the ego-machine and a second dynamic model indicating at least a second direction of travel of an object;

determine, based at least on the first direction of travel, the second direction of travel, and an assumption that the object will change from navigating in the second direction of travel to navigating in a third direction of travel to attempt to collide with the ego-machine, a perception zone associated with that includes an area of an environment that extends from a first location of the ego-machine and at least partially towards a second location of the object;

determine whether the object is located within the perception zone; and

cause the ego-machine to navigate based at least on the determination of whether the object is located within the perception zone.

17 . The one or more processors of claim 16 , wherein the one or more processors are comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing digital twin operations;

a system for performing real-time streaming;

a system for generating or presenting virtual reality (VR) content;

a system for generating or presenting augmented reality (AR) content;

a system for generating or presenting mixed reality (MR) content;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

18 . The one or more processors of claim 16 , wherein the processing circuitry is further to:

determine, based at least on whether the object is located within the perception zone, whether the object includes a safety-critical object,

wherein the ego-machine is caused to navigate based at least on the determination of whether the object includes the safety-critical object.

19 . The one or more processors of claim 16 , wherein the perception zone is further determined based at least on an assumption that the ego-machine will change from navigating in the first direction of travel to navigating in a fourth direction of travel to attempt to collide with the object.

20 . The one or more processors of claim 16 , wherein the perception zone is further determined based at least on an assumption that no other object is located between the ego-machine and the object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 13, 2022
From: TOPAN, SEVER IOAN; LEUNG, KAREN YAN MING; TUPEKAR, PRITISH; SCHMERLING, EDWARD FU; COX, MICHAEL; NILSSON, HANS JONAS; PAVONE, MARCO; CHEN, YUXIAO
To: NVIDIA CORPORATION
Reel/Frame 061075/0555 →
Continuity (1)
Related Publication 20240085914A1 · Mar 14, 2024
References Cited (30)
US 11370424B1 · Cohen · 2022 [cited by examiner]
US 12071157B1 · Prioletti · 2024 [cited by examiner]
US 20100324771A1 · Yabushita · 2010 [cited by examiner]
US 20200133281A1 · Gomez Gutierrez · 2020 [cited by examiner]
US 20210405638A1 · Boyraz · 2021 [cited by examiner]
ISO 26262, “Road vehicle—Functional safety,” International standard for functional safety of electronic system, https://en.wikipedia.org/wiki/ISO_26262, accessed on Sep. 13, 2021, 8 pgs. [cited by applicant]
Topan, Sever, et al.; “Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation”, Jun. 2022; 10 pgs. [cited by applicant]
Albus, James S.; “4D-RCS A Reference Model Architecture for Intelligent Unmanned Ground Vehicles” Unmanned Ground Vehicle Technology IV, vol. 4715, pp. 303-310, 2002, 8 pgs. [cited by applicant]
Volk, G., et al; “A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems”, In Proc. IEEE Int. Conf. on Intelligent Transportation Systems, 2020. [cited by applicant]
Bansal, Ayoosh, et al.; “Risk Ranked Recall: Collision Safety Metric for Object Detection Systems in Autonomous Vehicles”; https://arxiv.org/abs/2106.04146; Jun. 8, 2021, 4 pgs. [cited by applicant]
Mitchell, I.M.; et al.; “A Time-Dependent Hamilton-Jacobi Formulation of Reachable Sets for Continuous Dynamic Games”, IEEE Transactions on Automatic Control, vol. 50, No. 7, pp. 947-957, 2005. [cited by applicant]
Margellos, Kostas, et al.; “Hamilton-Jacobi Formulation for Reach-Avoid Differential Games”, IEEE Transaction on Automatic Control, vol. 56, No. 8, pp. 1849-1861, 2011. [cited by applicant]
Bansal, S., et al.; “Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances”; https://arxiv.org/abs/1709.07523; Sep. 21, 2017, 12 pgs. [cited by applicant]
Caesar, H., et al.: “NuScenes: A Multimodal Dataset for Autonomous Driving”; https://arxiv.org/abs/1903.11027; May 5, 2020, 16 pgs. [cited by applicant]
Dahl, J.; “Collision Avoidance: A Literature Review on Threat-Assessment Techniques”; IEEE Transactions on Intelligent Vehicles, vol. 4, No. 1, pp. 101-113, 2019. [cited by applicant]
Aravantinos, V., et al.; “Making the Relationship between Uncertainty Estimation and Safety Less Uncertain”; in Conf. on Design, Automation and Test in Europe, 2020. [cited by applicant]
Lyssenko, M., et al.; “From Evaluation to Verification: Towards Task-Oriented Relevance Metrics for Pedestrian Detection in Safety-Critical Domains”; in IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshop… [cited by applicant]
Hoss, M.; “A Review of Testing Object-Based Environment Perception for Safe Automated Driving”; https://arxiv.org/abs/2102.08460; Feb. 16, 2021, 23 pgs. [cited by applicant]
Philion, J., et al.; “Learning to Evaluate Perception Models Using Planner-Centric Metrics”; https://arxiv.org/abs/2004.08745; Apr. 19, 2020, 10 pgs. [cited by applicant]
Guo, Y., et al.; “The Efficacy of Neural Planning Metrics: A Meta-Analysis of PKL on NuScenes”; https://arxiv.org/abs/2010.09350; Jul. 13, 2021, 4 pgs. [cited by applicant]
Berk, M., et al.; “Assessing the Safety of Environment Perception in Automated Driving Vehicles”; SAE International Journal of Transportation Safety, vol. 8, No. 1, 2020. [cited by applicant]
Salay, R., et al.; “A Safety Analysis Method for Perceptual Components in Automated Driving”, in IEEE Int. Symp. On Software Reliability Engineering, 2019. [cited by applicant]
Leung, K., et al.; “Towards the Unification and Data-Driven Synthesis of Autonomous Vehicle Safety Concepts”; https://arxiv.org/abs/2107.14412, Jun. 20, 2022, 9 pgs. [cited by applicant]
Shalev-Shwartz, S., et al.; “On a Formal Model of Safe and Scalable Self-Driving Cars”; https://arxiv.org/abs/1708.06374, Oct. 27, 2018, 27 pgs. [cited by applicant]
Bajcsy, A., et al.; “An Efficient Reachability-Based Framework for Provably Safe Autonomous Navigation in Unknown Environments”; https://arxiv.org/abs/1905.00532, May 1, 2019, 10 pgs. [cited by applicant]
Leung, K., et al.; “On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions”; https://arxiv.org/abs/2012.03390, Dec. 6, 2020, 18 pgs. [cited by applicant]
Fisac, J.F.; et al.; “A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems”; https://arxiv.org/abs/1705.01292; Feb. 14, 2018, 16 pgs. [cited by applicant]
Schmerling, E.; “HJ Reachability in JAX” Available at https://github.com/StanfordASL/hj_reachability. [cited by applicant]
Vaskov, S., et al.; “Not-at-Fault Driving in Traffic: A Reachability-Based Approach”; in Proc. IEEE Int. Conf. on Intelligent Transportation Systems, 2019. [cited by applicant]
Zhu, B., et al.; “Class-Balanced Grouping and Sampling for Point Cloud 3D Object Detection”; https://arxiv.org/abs/1908.09492; Aug. 26, 2019, 8 pgs. [cited by applicant]