Virtual testing of autonomous environment control system
Methods and systems for assessing, detecting, and responding to malfunctions involving components of autonomous vehicles and/or smart homes are described herein. Autonomous operation features and related components can be assessed using direct or indirect data regarding operation. Such assessment may be performed to determine the robustness of autonomous systems, including the use of virtual assessment of software components within a simulated environment. To this end, a server may retrieve one or more routines associated with autonomous operation. The server may also generate a set of test data associated with test conditions. The server may also execute an emulator that virtually simulates autonomous environment. The test data may be presented to the routines executing in the emulator to generate output data. The server may then analyze the output data to determine a quality metric.
1. A computer-implemented method for evaluating smart home control software, comprising:
generating, by one or more processors, simulated sensor data for evaluating an autonomous operation feature of a plurality of autonomous operation features of a smart home operating system;
executing, by the one or more processors, an emulator program using the simulated sensor data, wherein the emulator program is configured to mimic the smart home operating system running on a smart home controller of a smart home;
recording, in the memory, output generated by the emulator program in response to the simulated sensor data, wherein the output is associated with the simulated sensor data; and
generating, by the one or more processors, a quality metric for the autonomous operation feature based upon the recorded output generated in response to the simulated sensor data, wherein the quality metric indicates one or more risk levels associated with use of the autonomous operation feature of the plurality of autonomous operation features.
2. The computer-implemented method of claim 1 , wherein the simulated sensor data comprises one or more sequences of simulated sensor signals associated with one or more time durations of simulated smart home operation.
3. The computer-implemented method of claim 2 , wherein the emulator program operates at a speed that is faster than a real-time speed at which the smart home operating system operates.
4. The computer-implemented method of claim 1 , wherein generating the quality metric includes:
comparing, by one or more processors, the recorded output against baseline output values associated with a set of test data.
5. The computer-implemented method of claim 4 , wherein generating the quality metric further includes:
determining, by one or more processors, one or more measures of differences between the recorded output and the baseline output values; and
determining, by one or more processors, the quality metric based upon the determined one or more measures of differences.
6. The computer-implemented method of claim 4 , wherein the baseline output values are associated with another version of smart home control software for the autonomous operation feature.
7. The computer-implemented method of claim 5 , wherein the determined one or more measures of differences include at least one indication of an improvement of the recorded output over the baseline output values.
8. The computer-implemented method of claim 1 , wherein a set of test data includes a plurality of subsets of test data, each subset associated with a combination of environmental conditions to be simulated for evaluation of the autonomous operation feature.
9. A computer system configured to evaluate smart home control software, comprising:
one or more processors; and
a non-transitory program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to:
generate simulated sensor data for evaluating an autonomous operation feature of a plurality of autonomous operation features of a smart home operating system;
execute an emulator program using the simulated sensor data, wherein the emulator program is configured to mimic the smart home operating system running on a smart home controller of a smart home;
record output generated by the emulator program in response to the simulated sensor data, wherein the output is associated with the simulated sensor data; and
generate a quality metric for the autonomous operation feature based upon the recorded output generated in response to the simulated sensor data, wherein the quality metric indicates one or more risk levels associated with use of the autonomous operation feature of the plurality of autonomous operation features.
10. The computer system of claim 9 , wherein the simulated sensor data comprises one or more sequences of simulated sensor signals associated with one or more time durations of simulated smart home operation.
11. The computer system of claim 9 , wherein the emulator program operates at a speed that is faster than a real-time speed at which the smart home operating system operates.
12. The computer system of claim 9 , wherein to generate the quality metric, the instructions, when executed, further cause the computer system to:
compare the recorded output against baseline output values associated with a set of test data;
determine one or more measures of differences between the recorded output and the baseline output values; and
determine the quality metric based upon the determined one or more measures of differences.
13. The computer system of claim 12 , wherein the baseline output values are associated with another version of smart home control software for the autonomous operation feature.
14. The computer system of claim 12 , wherein the determined differences include at least one indication of an improvement of the recorded output over the baseline output values.
15. A non-transitory computer-readable storage medium storing processor executable instructions, that when executed cause one or more processors to:
generate simulated sensor data for evaluating an autonomous operation feature of a plurality of autonomous operation features of a smart home operating system;
execute an emulator program using the simulated sensor data, wherein the emulator program is configured to mimic the smart home operating system running on a smart home controller of a smart home;
record output generated by the emulator program in response to the simulated sensor data, wherein the output is associated with the simulated sensor data; and
generate a quality metric for the autonomous operation feature based upon the recorded output generated in response to the simulated sensor data, wherein the quality metric indicates one or more risk levels associated with use of the autonomous operation feature of the plurality of autonomous operation features.