VIRTUAL TESTING OF AUTONOMOUS VEHICLE CONTROL SYSTEM
Methods and systems for assessing, detecting, and responding to malfunctions involving components of autonomous vehicles and/or smart homes are described herein. Autonomous operation features and related components can be assessed using direct or indirect data regarding operation to determine the robustness of autonomous systems, including the use of virtual assessment of software components within a simulated environment. A server may retrieve one or more routines associated with autonomous operation. The server may also generate a set of test data associated with test conditions. The server may also execute an emulator that virtually simulates autonomous environment. The test data may be presented to the routines executing in the emulator to generate output data. The server may then analyze the output data to determine a quality metric.
1 . A computer-implemented method for evaluating autonomous vehicle control software, comprising:
receiving, at one or more processors, a selection of a vehicle operating system;
receiving, via a user interface, a selection of one or more indicators of the test conditions, the test conditions comprising environmental conditions, the environmental conditions comprising at least one of: time of day, weather, road type, traffic, neighborhood, or construction;
generating, by one or more processors, a set of test data for evaluating an autonomous operation feature, the set of test data including simulated sensor data associated with the test conditions, wherein the simulated sensor data comprises one or more sequences of simulated sensor signals associated with one or more time durations of simulated vehicle operation;
executing, by one or more processors, an emulator program configured to mimic the selected vehicle operating system running on an on-board computer of an autonomous vehicle;
implementing, within the emulator program, one or more routines of the autonomous operation feature;
presenting, by one or more processors, the simulated sensor data of the set of test data to the one or more routines;
recording, in the memory, output generated by the one or more routines in response to the simulated sensor data; and
generating, by one or more processors, a quality metric for the autonomous operation feature based upon the recorded output.
2 . (canceled)
3 . (canceled)
4 . (canceled)
5 . The computer-implemented method of claim 1 , wherein:
the set of test data further includes simulated control data from one or more additional autonomous operation features;
the simulated control data is associated with the simulated sensor data; and
the simulated control data are presented to the one or more routines with the simulated sensor data.
6 . The computer-implemented method of claim 1 , further comprising presenting the simulated sensor data to the one or more routines, wherein at least a portion of the simulated sensor data presented to the one or more routines is selected for presentation based upon at least a portion of the output generated by the one or more routines in response to another portion of the simulated sensor data.
7 . The computer-implemented method of claim 1 , wherein generating the quality metric includes:
determining, by one or more processors, one or more measures of differences between the recorded output and the baseline output values; and
determining, by one or more processors, the quality metric based upon the determined one or more measure of differences.
8 . (canceled)
9 . The computer-implemented method of claim 7 , wherein the determined differences include at least one indication of an improvement of the recorded output over the baseline output values.
10 . The computer-implemented method of claim 1 , wherein the set of test data includes a plurality of subsets of test data, each subset associated with a combination of environmental conditions to be simulated for evaluation of the autonomous operation feature.
11 . The computer-implemented method of claim 1 , wherein the quality metric indicates one or more risk levels associated with the autonomous operation feature.
12 . A computer system configured to evaluate autonomous vehicle control software, comprising:
one or more processors; and
a non-transitory program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to:
receive a selection of a vehicle operating system;
receive, via a user interface, a selection of one or more indicators of the test conditions, the test conditions comprising environmental conditions, the environmental conditions comprising at least one of: time of day, weather, road type, traffic, neighborhood, or construction;
generate a set of test data for evaluating an autonomous operation feature, the set of test data including simulated sensor data associated with the test conditions, wherein the simulated sensor data comprises one or more sequences of simulated sensor signals associated with one or more time durations of simulated vehicle operation;
execute an emulator program configured to mimic the selected vehicle operating system running on an on-board computer of an autonomous vehicle;
implement, within the emulator program, one or more routines of the autonomous operation feature;
present the simulated sensor data of the set of test data to the one or more routines;
record output generated by the one or more routines in response to the simulated sensor data; and
generate a quality metric for the autonomous operation feature based upon the recorded output.
13 . (canceled)
14 . (canceled)
15 . (canceled)
16 . The computer system of claim 12 , wherein to generate the quality metric, the instructions, when executed, further cause the computer system to:
determine one or more measures of differences between the recorded output and the baseline output values; and
determine the quality metric based upon the determined one or more measure of differences.
17 . (canceled)
18 . The computer system of claim 16 , wherein the determined differences include at least one indication of an improvement of the recorded output over the baseline output values.
19 . The computer system of claim 12 , wherein the quality metric indicates one or more risk levels associated with the autonomous operation feature.
20 . A non-transitory computer-readable storage medium storing processor-executable instructions, that when executed cause one or more processors to:
receive a selection of a vehicle operating system;
receive, via a user interface, a selection of one or more indicators of the test conditions, the test conditions comprising environmental conditions, the environmental conditions comprising at least one of: time of day, weather, road type, traffic, neighborhood, or construction;
generate a set of test data for evaluating an autonomous operation feature, the set of test data including simulated sensor data associated with the test conditions, wherein the simulated sensor data comprises one or more sequences of simulated sensor signals associated with one or more time durations of simulated vehicle operation;
execute an emulator program configured to mimic the selected vehicle operating system running on an on-board computer of an autonomous vehicle;
implement, within the emulator program, one or more routines of the autonomous operation feature;
present the simulated sensor data of the set of test data to the one or more routines;
record output generated by the one or more routines in response to the simulated sensor data; and
generate a quality metric for the autonomous operation feature based upon the recorded output.
21 . The computer-implemented method of claim 1 , the quality metric being indicative of an effectiveness of the evaluated autonomous operation feature in controlling the autonomous vehicles.
22 . The computer system of claim 12 , the quality metric being indicative of an effectiveness of the evaluated autonomous operation feature in controlling the autonomous vehicles.
23 . The non-transitory computer-readable storage medium of claim 20 , the quality metric being indicative of an effectiveness of the evaluated autonomous operation feature in controlling the autonomous vehicles
24 . The computer-implemented method of claim 1 , wherein the quality metric is based upon a comparison of the recorded output generated by the one or more routines of the autonomous operation feature and baseline output values, and wherein the baseline output values are associated with another version of autonomous vehicle control software for the autonomous operation feature.
25 . The computer system of claim 12 , wherein the quality metric is based upon a comparison of the recorded output generated by the one or more routines of the autonomous operation feature and baseline output values, and wherein the baseline output values are associated with another version of autonomous vehicle control software for the autonomous operation feature.
26 . The non-transitory computer-readable storage medium of claim 20 , wherein the quality metric is based upon a comparison of the recorded output generated by the one or more routines of the autonomous operation feature and baseline output values, and wherein the baseline output values are associated with another version of autonomous vehicle control software for the autonomous operation feature.
27 . The computer-implemented method of claim 1 , wherein the emulator program operates at an accelerated speed to present the one or more sequences of simulated sensor data to the one or more routines in one or more actual time durations that are less than the one or more time durations of simulated vehicle operation.
28 . The computer system of claim 12 , wherein the emulator program operates at an accelerated speed to present the one or more sequences of simulated sensor data to the one or more routines in one or more actual time durations that are less than the one or more time durations of simulated vehicle operation.
29 . The non-transitory computer-readable storage medium of claim 20 , wherein the emulator program operates at an accelerated speed to present the one or more sequences of simulated sensor data to the one or more routines in one or more actual time durations that are less than the one or more time durations of simulated vehicle operation.