IP Library › Granted Patent US 12,566,905
Granted Patent B2
US 12,566,905 · App. 18/505,672 · Granted Mar 3, 2026

Virtual environment scenarios and observers for autonomous machine applications

Inventors: Ahmed Nassar (San Jose, CA); Justyna Zander (Cupertino, CA); David Auld (Saratoga, CA)
Assignee: NVIDIA Corporation
G06F30/27B60W50/00G05D1/0088B60W2050/0083B60W60/001G06N3/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,566,905
App. No.
18/505,672
Granted
Mar 3, 2026
Kind
B2
Abstract

In various examples, scenarios may be defined using a declarative description—e.g., defining a behavior of interest—that the present system may convert into a procedural description for generating one or more instances and/or variations of a scenario for testing an autonomous or semi-autonomous machine in a virtual environment. The system may execute observers or evaluators for testing the performance and accuracy of the machine and may compute coverage of various elements based on the generated virtual scenarios, and may feed the results back to the system to generate additional instances and/or variations where the coverage or accuracy is below a desired level. As a result, the system may include an end-to-end framework for generating scenarios in virtual environments, testing and validating the scenarios themselves, and/or testing and validating the underlying autonomous or semi-autonomous systems of the machine—all based on a declarative description.

Claims (80)

1 . A method comprising:

receiving a declarative description of a desired behavior to be tested;

determining, using one or more ontologies related to one or more simulated environments and based at least on the declarative description, one or more commands for generating one or more scenarios related to the desired behavior within a simulation system;

generating, using the simulation system and based at least on the one or more commands, the one or more scenarios representing one or more simulated variations of the desired behavior as being performed using one or more machines;

performing the one or more simulated variations represented by the one or more scenarios within the simulation system;

determining, based at least on the one or more simulated variations, one or more accuracies indicating whether the one or more scenarios actually represent the desired behavior; and

determining, based at least on the one or more accuracies, a coverage value indicating a number of the one or more scenarios that actually represent the desired behavior.

2 . The method of claim 1 , further comprising:

determining that the coverage value is below a threshold;

based at least on the coverage value being below the threshold, generating one or more second commands for generating one or more second scenarios related to the observable behavior; and

determining, based at least on the one or more second scenarios, an updated coverage value associated with the desired observable.

3 . The method of claim 2 , wherein:

the one or more commands are associated with at least one of path information or dynamic actor information associated with the one or more scenarios; and

the generating the one or more second commands is further based at least on the at least one of the path information or the dynamic actor information.

4 . The method of claim 1 , further comprising:

determining that the coverage value is equal to or greater than a threshold; and

determining, based at least on the coverage value being equal to or greater than the threshold, to refrain from generating one or more second scenarios related to the desired observable behavior.

5 . The method of claim 1 , further comprising:

for at least one scenario of the one or more scenarios that does not satisfy the desired behavior, analyzing the at least one scenario to determine criticality of the at least one scenario for testing the one or more machines; and

based at least on the analyzing, determining to generate one or more second commands for generating one or more second scenarios related to the desired behavior.

6 . The method of claim 1 , further comprising:

testing at least one feature of the one or more machines against the one or more scenarios; and

updating the at least one feature of the one or more machines based at least in part on the testing.

7 . The method of claim 6 , wherein the at least one feature includes at least one of a hardware component, a software component, or a sensor model.

8 . The method of claim 1 , wherein the determining the coverage value comprises:

determining, based at least on the one or more accuracies, the number of the one or more scenarios that actually represent the desired behavior; and

determining, based at least on the number of the one or more scenarios, a percentage of the one or more scenarios that actually represent the desired behavior, the coverage value including the percentage.

9 . A system comprising:

one or more processors to:

receive a declarative description associated with a desired observable;

determine, based at least on the declarative description, one or more commands for generating one or more scenarios related to the desired observable;

generate, based at least in part on the one or more commands, the one or more scenarios that represent one or more simulation variations of the desired observable being performed using one or more machines;

perform one or more simulations associated with the one or more scenarios; and

determine, based at least on the one or more simulations, a coverage value indicating a percentage of the one or more scenarios that actually represent the desired observable.

10 . The system of claim 9 , wherein the one or more processors are further to:

determine one or more accuracies associated with the one or more scenarios actually representing the desired observable,

wherein the determination of the coverage value is based at least on the one or more accuracies.

11 . The system of claim 9 , wherein the one or more processors are further to:

determine that the coverage value is below a threshold;

based at least on the coverage value being below the threshold, generate one or more second commands for generating one or more second scenarios related to the desired observable;

perform one or more second simulations associated with the one or more second scenarios; and

determine, based at least on the one or more second simulations, an updated coverage value associated with the desired observable.

12 . The system of claim 11 , wherein:

the one or more commands are associated with at least one of path information or dynamic actor information associated with the one or more scenarios; and

the generation of the one or more second commands is further based at least on the at least one of the path information or the dynamic actor information.

13 . The system of claim 9 , wherein the one or more processors are further to:

determine that the coverage value is equal to or greater than a threshold; and

determine, based at least on the coverage value being equal to or greater than the threshold, to refrain from generating one or more second scenarios related to the desired observable.

14 . The system of claim 9 , wherein the one or more processors are further to:

for at least one scenario of the one or more scenarios that does not satisfy the desired observable, analyze the scenario to determine criticality of the scenario for testing the one or more machines; and

based at least on the analysis, determine to generate one or more second commands for generating one or more second scenarios related to the desired observable.

15 . The system of claim 9 , wherein the one or more processors are further to:

test at least one feature of the one or more machines against the one or more scenarios; and

update the at least one feature of the one or more machines.

16 . The system of claim 9 , wherein the generation of the one or more scenarios comprises:

determining, based at least on the one or more commands, one or more metrics for generating the one or more simulation variations; and

generating, based at least on the one or more metrics, the one or more scenarios that represent the one or more simulation variations of the desired observable being performed using the one or more machines.

17 . The system of claim 9 , wherein the one or more scenarios are further generated based at least on one or more ontologies, the one or more ontologies including at least one of a domain ontology, an actor ontology, a map ontology, an environment ontology, or a control ontology.

18 . The system of claim 9 , wherein the system is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing deep learning operations;

a system implemented using a robot;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

19 . One or more processors comprising processing circuitry to:

determine a coverage value indicating at least one of a number of one or more scenarios or a percentage of the one or more scenarios that represent a desirable observable, wherein the coverage value is determined based at least on performing one or more simulation variations associated with the one or more scenarios that are generated using a declarative description associated with the desired observable.

20 . The one or more processors of claim 19 , wherein the one or more processors are comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing deep learning operations;

a system implemented using a robot;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 9, 2023
From: NASSAR, AHMED; ZANDER, JUSTYNA; AULD, DAVID
To: NVIDIA CORPORATION
Reel/Frame 065512/0264 →
Continuity (2)
Continuation 16824202 · Mar 19, 2020
Related Publication 20240078363A1 · Mar 7, 2024
References Cited (59)
US 10157331B1 · Tang et al. · 2018 [cited by applicant]
US 20040252864A1 · Chang et al. · 2004 [cited by applicant]
US 20070182528A1 · Breed et al. · 2007 [cited by applicant]
US 20160247290A1 · Liu et al. · 2016 [cited by applicant]
US 20170010108A1 · Shashua · 2017 [cited by applicant]
US 20170344808A1 · El-Khamy et al. · 2017 [cited by applicant]
US 20170364083A1 · Yang et al. · 2017 [cited by applicant]
US 20170371340A1 · Cohen et al. · 2017 [cited by applicant]
US 20190129831A1 · Goldberg · 2019 [cited by applicant]
US 20190155291A1 · Heit · 2019 [cited by examiner]
US 20190258878A1 · Koivisto et al. · 2019 [cited by applicant]
US 20190303759A1 · Farabet · 2019 [cited by examiner]
US 20200134494A1 · Venkatadri · 2020 [cited by examiner]
US 20200409369A1 · Zaytsev et al. · 2020 [cited by applicant]
US 20200410063A1 · O'Malley · 2020 [cited by applicant]
US 20210116915A1 · Jiang et al. · 2021 [cited by applicant]
EP 1930863A2 · 2008 [cited by applicant]
EP 2384009A2 · 2011 [cited by applicant]
WO 2018002910A1 · 2018 [cited by applicant]
Wei Chen, Leila Kloul. An Ontology-based Approach to Generate the Advanced Driver Assistance Use Cases of Highway Traffic. 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge … [cited by applicant]
Nassar, Ahmed; Non-Final Office Action for U.S. Appl. No. 16/824,202, filed Mar. 19, 2020, mailed Aug. 26, 2022, 38 pgs. [cited by applicant]
Nassar, Ahmed; International Preliminary Report on Patentability for PCT Application No. PCT/US2021/023343, filed Mar. 19, 2021, mailed Sep. 29, 2022, 12 pgs. [cited by applicant]
Garnett, N., et al., (2017). “Real-time category-based and general obstacle detection for autonomous driving”. In Proceedings of the IEEE International Conference on Computer Vision Workshops (pp. 198-205). [cited by applicant]
Zhong, Yiran et al; “Self-Supervised Learning for Stereo Matching with Self-Improving Ability”, Arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Sep. 4, 2017. 13 pgs. [cited by applicant]
Pang, Jiahao et al; “Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching”, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), IEEE, Oct. 22, 2017, pp. 878-886. 9… [cited by applicant]
Godard, Clement et al: “Unsupervised Monocular Depth Estimation with Left-Right Consistency”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Proceedings, IEEE Computer Society, US, Jul. 21,… [cited by applicant]
International Search Report and Written Opinion mailed Nov. 7, 2019 in International Patent Application No. PCT/US2019/022753, 22 pgs. [cited by applicant]
Kendall, Alex et al., “End-to-End Learning of Geometry and Context for Deep Stereo Regression” ARXIV.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Mar. 13, 2017, 10 pgs. [cited by applicant]
International Search Report and Written Opinion mailed Oct. 17, 2019 in International Patent Application No. PCT/US2019/012535, 24 pgs. [cited by applicant]
International Search Report and Written Opinion mailed Jul. 25, 2019 in International Patent Application No. PCT/US2019/018348, 22 pgs. [cited by applicant]
Arvind Jayarahan et al: “Creating 3D Virtual Driving Environments for Simulation-Aided Development of Autonomous Driving and Active Safety”, SAE Technical Paper Series, vol. 1, Mar. 28, 2017 (Mar. 28, 2017), XP055518353… [cited by applicant]
International Search Report and Written Opinion mailed Jun. 26, 2019 in International Patent Application No. PCT/US2019/024400, 15 pgs. [cited by applicant]
International Search Report and Written Opinion mailed Aug. 26, 2019 in International Patent Application No. PCT/US2019/022592, 18 pgs. [cited by applicant]
Bach, M., et al., “Multi-camera traffic light recognition using a classifying Labeled Multi-Bernoulli filter”, IEEE Intelligent Vehicles Symposium (IV), pp. 1045-1051 (Jun. 2017). [cited by applicant]
Bojarski, M., et al., “End to End Learning for Self-Driving Cars”, arXiv: 1604.07316v1 [cs.CV], XP055570062, Retrieved from the Internet URL:https://nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-e… [cited by applicant]
Liu, H., et al., “Neural Person Search Machines”, IEEE International Conference on Computer Vision (ICCV), pp. 493-501 (2017). [cited by applicant]
Rothe, R., et al., “Non-maximum Suppression for Object Detection by Passing Messages Between Windows”, ETH Library, pp. 1-17 (2015). [cited by applicant]
Tao, A., “Detectnet: Deep neural network for object detection in digits”, NVIDIA Developer Blog, Retrieved from Internet URL: https://devblogs.nvidia.com/detectnet-deep-neural-network-object-detection-digits/, accessed … [cited by applicant]
Weber, M., et al., “DeepTLR: A single deep convolutional network for detection and classification of traffic lights”, IEEE Intelligent Vehicles Symposium (IV), pp. 8 (Jun. 2016). [cited by applicant]
“System and Method for Training, Testing, Verifying, and Validating Autonomous and Semi-Autonomous Vehicles”, U.S. Appl. No. 62/648,399, filed Mar. 27, 2018. [cited by applicant]
Nassar, Ahmed; Final Office Action for U.S. Appl. No. 16/824,202, filed Mar. 19, 2020, mailed Jan. 24, 2023, 36 pgs. [cited by applicant]
Nassar, Ahmed; Non-Final Office Action for U.S. Appl. No. 16/824,202, filed Mar. 19, 2020, mailed Apr. 5, 2023, 33 pgs. [cited by applicant]
Nassar, et al.; Final Office Action for U.S. Appl. No. 16/824,202, filed Mar. 19, 2020, mailed Aug. 9, 2023, 28 pgs. [cited by applicant]
Bagschik, et al.; “Ontology based Scene Creation for the Development of Automated Vehicles,” 2018 IEEE Intelligent Vehicles Symposium (IV), Jun. 2018, 8 pgs. [cited by applicant]
Amersback, et al.; “Defining Required and Feasible Test Coverage for Scenario-Based Validation of Highly Automated Vehicles,” IEEE Intelligent Transportation Systems Conference (ITSC), 2019, 5 pgs. [cited by applicant]
Majzik, et al.; “Towards System-Level Testing with Coverage Guarantees for Autonomous Vehicles,” ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS), 2019, 6 pgs. [cited by applicant]
Menzell, et al.; “From Functional to Logical Scenarios: Detailing a Keyword-Based Scenario Description for Execution in a Simulation Environment,” IEEE Intelligent Vehicles Symposium (IV), 2019, 8 pgs. [cited by applicant]
Menzell, et al.; “Scenarios for Development, Test and Validation of Automated Vehicles,” IEEE Intelligent Vehicles Symposium (IV), 2018, 7 pgs. [cited by applicant]
Nassar, Ahmed; International Search Report and Written Opinion for PCT Application No. PCT/US2021/023343, filed Mar. 19, 2021, mailed Jul. 26, 2021, 13 pgs. [cited by applicant]
Fox, et al.; “PDDL2. 1: An Extension to PDDL for Expressing Temporal Planning Domains,” Journal of Artificial Intelligence Research, 2003, 64 pgs. [cited by applicant]
Gerevini, et al.; “Deterministic planning in the fifth international planning competition: PDDL3 and experimental evaluation of the planners” 2009, 50 pgs. [cited by applicant]
Accellera Systems Initiative, IEEE Standard for Standard SystemC Language Reference Manual. Chapter 4 and 5. IEEE Std (Revision of IEEE Std 1666-2005) 2012, 638 pgs. [cited by applicant]
Annex E., of IEEE Standard for SystemVerilog Unified Hardware Design, Specification; and Verification Language. IEEE Std 1800TM-2005, 2005, 1315 pgs. [cited by applicant]
Shoham, et al.; “Multiagent systems: Algorithmic, Game-Theoretic, and Logical Foundations,” Cambridge University Press, (2008) 532 pgs. [cited by applicant]
LaValle, et al.; “Planning Algorithms. Cambridge University Press, 2006,” Section 7.6, Chapters 5,8, and 10, (2006) 1023 pgs. [cited by applicant]
Sutton, et al.; “Reinforcement learning: An introduction.” MIT Press, 2018, 10 pgs. [cited by applicant]
Nassar, Ahmed; First Office Action for Chinese Patent Application No. 202180008425.5, filed Jul. 6, 2022, mailed Dec. 23, 2024, 54 pgs. [cited by applicant]
Nassar, Ahmed; Second Office Action for Chinese Patent Application No. 202180008425.5, filed Jul. 6, 2022, mailed Jul. 3, 2025, 24 pgs. [cited by applicant]
Nassar, Ahmed; Decision on Rejection for Chinese Patent Application No. 202180008425.5, filed Jul. 6, 2022, mailed Sep. 23, 2025, 27 pgs. [cited by applicant]