Granted Patent B1

US 10,599,546 · App. 15/807,893 · Granted Mar 24, 2020

Inventors: Charly Mario Walther (San Francisco, CA); Nathan Steven Appelson (Pittsburgh, PA); Jeffrey Allen Howell (Pittsburgh, PA)

Assignee: UATC, LLC

G06F11/3457G06F11/3013

Abstract

Claims (62)

1. A computer-implemented method, comprising:

obtaining, by a computing system that comprises one or more computing devices, first data indicative of a test of an autonomous vehicle computing system;

determining, by the computing system, one or more autonomous vehicle capabilities that are tested by the test;

determining, by the computing system, a testing scenario that corresponds to the test, wherein the testing scenario is generated at least in part using real-world data;

obtaining, by the computing system, second data indicative of one or more user inputs provided via a first user interface, wherein the one or more user inputs are indicative of an association between the first data indicative of the test with third data indicative of the testing scenario and fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test;

in response to obtaining the second data indicative of the one or more user inputs, associating, by the computing system, the first data indicative of the test with the fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test and the third data indicative of the testing scenario;

storing, by the computing system in an accessible memory, the first data indicative of the test associated with the third data indicative of the testing scenario and the fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test;

obtaining, by the computing system, a query indicative of at least one of the test, the testing scenario, or at least one of the one or more autonomous vehicle capabilities; and

providing for display, by the computing system via one or more display devices, a second user interface that presents fifth data indicative of at least one of the test, the testing scenario, or the one or more autonomous vehicle capabilities.

2. The computer-implemented method of claim 1 , wherein the real-world data comprises driving log data generated by one or more autonomous vehicles.

3. The computer-implemented method of claim 1 , wherein the real-world data comprises sensor data obtained via one or more sensors of one or more autonomous vehicles, wherein the sensor data comprises at least one of image data, Light Detection and Ranging data, or Radio Detection and Ranging data.

4. The computer-implemented method of claim 3 , wherein the image data comprises video data.

5. The computer-implemented method of claim 1 , wherein the method further comprises:

identifying, by the computing system, one or more testing parameters based at least in part on the real-world data; and

generating, by the computing system, the third data indicative of the testing scenario based at least in part on the one or more testing parameters.

6. The computer-implemented method of claim 5 , wherein the testing parameters are indicative of at least one of a geographic area or one or more objects.

7. The computer-implemented method of claim 1 , wherein the second data indicative of one or more user inputs comprises sixth data indicative of a first user input that identifies the testing scenario as corresponding to the test; and

wherein determining, by the computing system, the third data indicative of the testing scenario that corresponds to the first data indicative of the test comprises:

determining, by the computing system, that the testing scenario corresponds to the test based at least in part on the sixth data indicative of the first user input.

8. The computer-implemented method of claim 1 , wherein the second data indicative of one or more user inputs comprises seventh data indicative of a second user input that identifies the one or more autonomous vehicle capabilities; and

wherein determining, by the computing system, the fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test comprises:

determining, by the computing system, that the one or more autonomous vehicle capabilities are tested by the test based at least in part on the seventh data indicative of the second user input.

9. The computer-implemented method of claim 8 , wherein the one or more autonomous vehicle capabilities are included in a group of pre-defined autonomous vehicle capabilities.

10. The computer-implemented method of claim 1 , further comprising:

obtaining, by the computing system, test result data associated with the test, wherein the test result data is based at least in part on feedback data from the autonomous vehicle computing system;

associating, by the computing system, the test result data with the first data indicative of the test; and

storing, by the computing system in the accessible memory, the first data indicative of the test associated with the test result data.

11. A computing system for autonomous vehicle testing, comprising:

one or more processors;

one or more display devices; and

one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations comprising:

obtaining first data indicative of a test of an autonomous vehicle computing system;

determining one or more autonomous vehicle capabilities that are tested by the test;

determining a testing scenario that corresponds to the test, wherein the testing scenario is generated at least in part using real-world data;

obtaining second data indicative of one or more user inputs provided via a first user interface, wherein the one or more user inputs are indicative of an association between the first data indicative of the test with third data indicative of the testing scenario and fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test;

in response to obtaining the second data indicative of the one or more user inputs, associating the first data indicative of the test with the fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test and the third data indicative of the testing scenario;

storing, in an accessible memory, the first data indicative of the test associated with the third data indicative of the testing scenario and the fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test;

obtaining a query indicative of at least one of the test, the testing scenario, or at least one of the one or more autonomous vehicle capabilities; and

providing, for display via the one or more display devices, a second user interface that presents fifth data indicative of at least one of the test, the testing scenario, and the one or more autonomous vehicle capabilities.

12. The computing system of claim 11 , wherein the operations further comprise:

obtaining data indicating that the test is sound or unsound; and

designating, in the accessible memory, the test as sound or unsound based at least in part on the data indicating that the test is sound or unsound.

13. The computing system of claim 11 , wherein the operations further comprise:

obtaining data indicating that the test is in-scope or out-of-scope; and

designating, in the accessible memory, the test as in-scope or out-of-scope based at least in part on the data indicating that the test is in-scope or out-of-scope.

14. The computing system of claim 11 , wherein the real-world data comprises at least one of driving log data generated by one or more autonomous vehicles or sensor data obtained by the one or more autonomous vehicles.

15. The computing system of claim 11 , wherein the test comprises deploying an autonomous vehicle within a test track environment.

16. The computing system of claim 11 , wherein the operations further comprise:

obtaining test result data associated with the test, wherein the test result data is based at least in part on feedback data from the autonomous vehicle computing system; and

associating the test result data with the first data indicative of the test.

17. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations comprising:

obtaining first data indicative of a test of an autonomous vehicle computing system;

determining one or more autonomous vehicle capabilities that are tested by the test;

determining a testing scenario that corresponds to the test, wherein the testing scenario is generated at least in part using real-world data;

obtaining second data indicative of one or more user inputs provided via a first user interface, wherein the one or more user inputs are indicative of an association between the first data indicative of the test with third data indicative of the testing scenario and fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test;

in response to obtaining the second data indicative of the one or more user inputs, associating the first data indicative of the test with the fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test and the third data indicative of the testing scenario;

storing, in an accessible memory, the first data indicative of the test associated with the third data indicative of the testing scenario and the fourth data indicative of the one or more autonomous vehicle capabilities that are tested by the test;

obtaining a query indicative of at least one of the test, the testing scenario, or the one or more autonomous vehicle capabilities; and

providing, for display via one or more display devices, a second user interface that presents fifth data indicative of at least one of the test, the testing scenario, or the one or more autonomous vehicle capabilities.

18. The one or more tangible, non-transitory computer-readable media of claim 17 , wherein the operations further comprise:

obtaining data indicative of user input directed to initiating the test; and

causing the test to initiate based at least in part on the data indicative of the user input.

Assignments (3)

Continuity (1)

Cited By (12)

US 12,248,412 US 12,266,148 US 12,321,258 US 12,353,213 US 12,384,410 US 12,499,363 US 12,573,187 US 12,573,247 US 12,585,847 US 12,630,105 US 12,654,723 US 12,704,438