Virtual testing of autonomous environment control system
Methods and systems for assessing, detecting, and responding to malfunctions involving components of autonomous vehicles and/or smart homes are described herein. Autonomous operation features and related components can be assessed using direct or indirect data regarding operation. Such assessment may be performed to determine the robustness of autonomous systems, including the use of virtual assessment of software components within a simulated environment. To this end, a server may retrieve one or more routines associated with autonomous operation. The server may also generate a set of test data associated with test conditions. The server may also execute an emulator that virtually simulates autonomous environment. The test data may be presented to the routines executing in the emulator to generate output data. The server may then analyze the output data to determine a quality metric.
1. A computer-implemented method for evaluating smart home control software, comprising:
generating, by one or more processors, a set of test data for evaluating an autonomous operation feature of a plurality of autonomous operation features of a smart home operating system, the set of test data including simulated sensor data associated with test conditions;
executing, by the one or more processors, an emulator program using the simulated sensor data of the set of test data, wherein the emulator program is configured to mimic the smart home operating system running on a smart home controller of a smart home;
recording, in the memory, output generated by the emulator program in response to the simulated sensor data, wherein the output is associated with the simulated sensor data; and
generating, by the one or more processors, a quality metric for the autonomous operation feature based upon the recorded output generated in response to the simulated sensor data, wherein the quality metric indicates one or more risk levels associated with use of the autonomous operation feature of the plurality of autonomous operation features.
2. The computer-implemented method of claim 1 , wherein the simulated sensor data comprises one or more sequences of simulated sensor signals associated with one or more time durations of simulated smart home operation.
3. The computer-implemented method of claim 2 , wherein the emulator operates at a speed that is faster than a real-time speed at which the smart home operating system operates.
4. The computer-implemented method of claim 1 , wherein generating the set of test data includes:
receiving, via a user interface, a selection of one or more indicators of the test conditions, and
retrieving, from the memory, the simulated sensor data to form the set of test data based upon the one or more indicators of the test conditions.
5. The computer-implemented method of claim 1 , wherein:
the set of test data further includes simulated control data from one or more additional autonomous operation features;
the simulated control data is associated with the simulated sensor data; and
the simulated control data are used by the emulator program to mimic the smart home operating system.
6. The computer-implemented method of claim 1 , wherein at least a portion of the simulated sensor data.
7. The computer-implemented method of claim 1 , wherein generating the quality metric includes:
comparing, by one or more processors, the recorded output against baseline output values associated with the set of test data;
determining, by one or more processors, one or more measures of differences between the recorded output and the baseline output values; and
determining, by one or more processors, the quality metric based upon the determined one or more measure of differences.
8. The computer-implemented method of claim 7 , wherein the baseline output values are associated with another version of smart home control software for the autonomous operation feature.
9. The computer-implemented method of claim 7 , wherein the determined differences include at least one indication of an improvement of the recorded output over the baseline output values.
10. The computer-implemented method of claim 1 , wherein the set of test data includes a plurality of subsets of test data, each subset associated with a combination of environmental conditions to be simulated for evaluation of the autonomous operation feature.
11. A computer system configured to evaluate smart home control software, comprising:
one or more processors; and
a non-transitory program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to:
generate a set of test data for evaluating an autonomous operation feature of a plurality of autonomous operation features of a smart home operating system, the set of test data including simulated sensor data associated with test conditions;
execute an emulator program using the simulated sensor data of the set of test data, wherein the emulator program is configured to mimic the smart home operating system running on a smart home controller of a smart home;
record output generated by the emulator program in response to the simulated sensor data, wherein the output is associated with the simulated sensor data; and
generate a quality metric for the autonomous operation feature based upon the recorded output generated in response to the simulated sensor data, wherein the quality metric indicates one or more risk levels associated with use of the autonomous operation feature of the plurality of autonomous operation features.
12. The computer system of claim 11 , wherein the simulated sensor data comprises one or more sequences of simulated sensor signals associated with one or more time durations of simulated smart home operation.
13. The computer system of claim 12 , wherein the emulator operates at a speed that is faster than a real-time speed at which the smart home operating system operates.
14. The computer system of claim 11 , wherein to generate the set of test date, the instructions, when executed, further cause the computer system to:
receive, via a user interface, a selection of one or more indicators of the test conditions, and
retrieve, from the memory, the simulated sensor data to form the set of test data based upon the one or more indicators of the test conditions.
15. The computer system of claim 11 , wherein to generate the quality metric, the instructions, when executed, further cause the computer system to:
compare the recorded output against baseline output values associated with the set of test data;
determine one or more measures of differences between the recorded output and the baseline output values; and
determine the quality metric based upon the determined one or more measure of differences.
16. The computer system of claim 15 , wherein the baseline output values are associated with another version of smart home control software for the autonomous operation feature.
17. The computer system of claim 15 , wherein the determined differences include at least one indication of an improvement of the recorded output over the baseline output values.
18. A non-transitory computer-readable storage medium storing processor executable instructions, that when executed cause one or more processors to:
generate a set of test data for evaluating an autonomous operation feature of a plurality of autonomous operation features of a smart home operating system, the set of test data including simulated sensor data associated with test conditions;
execute an emulator program using the simulated sensor data of the set of test data, wherein the emulator program is configured to mimic the smart home operating system running on a smart home controller of a smart home;
record output generated by the emulator program in response to the simulated sensor data, wherein the output is associated with the simulated sensor data; and
generate a quality metric for the autonomous operation feature based upon the recorded output generated in response to the simulated sensor data, wherein the quality metric indicates one or more risk levels associated with use of the autonomous operation feature of the plurality of autonomous operation features.