Autonomous vehicle operation feature evaluation
Methods and systems for testing the operation of autonomous or semi-autonomous operation features in a virtual test environment are provided. Computer-executable instructions for implementing the features may be received and executed on a test device. Test input signals may be presented to software routines associated with the features, which may generate output signals including control commands. The output signals may be used to predict the response of a vehicle in the virtual test environment, which may include a simulation of vehicle responses to the output signals. Measures of the effectiveness of the features may be determined based upon the predicted responses of the vehicle, which may then be used to determine risk levels associated with the features.
1. A computer-implemented method for testing the effectiveness of one or more autonomous operation features for controlling vehicles by simulating operation of the one or more autonomous operation features in a virtual vehicle in a plurality of virtual test environments associated with a plurality of environmental conditions, comprising:
receiving, at one or more processors, a set of computer-readable instructions for implementing the one or more autonomous operation features, wherein the set of computer-readable instructions comprises one or more software routines configured to receive one or more input signals from at least one sensor and generate one or more output signals for controlling a vehicle;
receiving, at one or more processors of a sensor input simulator, the plurality of virtual test environments;
for each virtual test environment of the plurality of virtual test environments, automatically testing the one or more autonomous operation features without input from a user by:
generating, by one or more processors of the sensor input simulator, a fully virtual vehicle simulation including the virtual vehicle within the virtual test environment;
executing, by the one or more processors, the one or more software routines to respond to test input signals for the virtual vehicle within the fully virtual vehicle simulation;
receiving, at the one or more processors from the sensor input simulator, a plurality of test input signals for the virtual test environment, wherein the test input signals simulate the one or more signals from at least one sensor corresponding to a test scenario associated with the virtual test environment;
generating, by the one or more software routines executing on the one or more processors, one or more test output signals for the virtual vehicle in response to the received test input signals for the virtual test environment;
predicting, by the one or more processors of the sensor input simulator, one or more responses of the virtual vehicle in the virtual test environment to the one or more test output signals;
determining, by one or more processors of a server, a measure of the effectiveness of the one or more autonomous operation features based at least in part upon the one or more predicted responses of the virtual vehicle to the one or more test output signals associated with the plurality of virtual test environments; and
adjusting, by the one or more processors of the server, a risk profile associated with a vehicle having the one or more autonomous operation features based upon the determined measure of effectiveness by updating the risk profile to include one or more risk levels associated with autonomous operation of the vehicle by the one or more autonomous operation features.
2. The computer-implemented method of claim 1 , wherein the sensor input simulator accesses the one or more test input signals from a database containing a plurality of test signals.
3. The computer-implemented method of claim 1 , wherein generating the fully virtual vehicle simulation includes:
generating, by the one or more processors of the sensor input simulator, a simulation of the virtual vehicle in the virtual test environment;
determining, by the one or more processors of the sensor input simulator, simulated sensor data associated with the virtual vehicle in the virtual test environment; and
determining, by the one or more processors of the sensor input simulator, the one or more test input signals based at least in part upon the simulated sensor data.
4. The computer-implemented method of claim 1 , wherein the measure of the effectiveness of the one or more autonomous operation features includes one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features.
5. The computer-implemented method of claim 4 , wherein determining the one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features includes predicting, by the one or more processors of the server, the one or more risk levels based at least in part upon a comparison of (i) the one or more test output signals generated by the one or more software routines, (ii) one or more other test output signals generated by one or more other software routines of one or more other autonomous operation features in response to one or more other test input signals, and (iii) observed operating data regarding the one or more other autonomous operation features disposed within a plurality of other vehicles operating outside the plurality of virtual test environments.
6. The computer-implemented method of claim 5 , wherein the observed operating data regarding the one or more other autonomous operation features includes data regarding actual losses associated with insurance policies covering the plurality of other vehicles having the one or more other autonomous operation features.
7. The computer-implemented method of claim 1 , wherein each of the plurality of virtual test environments is based at least in part upon observed data regarding actual environments recorded by one or more sensors communicatively connected to a plurality of vehicles operating outside the plurality of virtual test environments.
8. A computer system for testing the effectiveness of one or more autonomous operation features for controlling vehicles by simulating operation of the one or more autonomous operation features in a virtual vehicle in a plurality of virtual test environments associated with a plurality of environmental conditions, comprising:
one or more processors;
a sensor input simulator configured to receive the plurality of virtual test environments and, for each virtual test environment: (i) generate a fully virtual vehicle simulation including the virtual vehicle within the virtual test environment and (ii) predict responses of the virtual vehicle within the virtual test environment; and
a non-transitory program memory coupled to the one or more processors and storing executable instructions that when executed by the one or more processors cause the computer system to:
receive a set of computer-readable instructions for implementing the one or more autonomous operation features, wherein the set of computer-readable instructions comprises one or more software routines configured to receive one or more input signals from at least one sensor and generate one or more output signals for controlling a vehicle;
for each virtual test environment of the plurality of virtual test environments, automatically testing the one or more autonomous operation features without input from a user by:
execute the one or more software routines to respond to test input signals for the virtual vehicle within the fully virtual vehicle simulation;
receive a plurality of test input signals for the virtual test environment from the sensor input simulator, wherein the test input signals simulate the one or more signals from at least one sensor corresponding to a test scenario associated with the virtual test environment;
generate one or more test output signals for the virtual vehicle using the one or more software routines in response to the received test input signals for the virtual test environment;
cause the sensor input simulator to predict one or more responses of the virtual vehicle in the virtual test environment to the one or more test output signals;
determine a measure of the effectiveness of the one or more autonomous operation features based at least in part upon the one or more predicted responses of the virtual vehicle to the one or more test output signals associated with the plurality of virtual test environments; and
adjust a risk profile associated with a vehicle having the one or more autonomous operation features based upon the determined measure of effectiveness by updating the risk profile to include one or more risk levels associated with autonomous operation of the vehicle by the one or more autonomous operation features.
9. The computer system of claim 8 , wherein the the sensor input simulator is further configured to access the one or more test input signals from a database containing a plurality of test signals.
10. The computer system of claim 8 , wherein the sensor input simulator is further configured to, for each virtual test environment:
generate a simulation of the virtual vehicle in the virtual test environment;
determine simulated sensor data associated with the virtual vehicle in the virtual test environment; and
determine the one or more test input signals based at least in part upon the simulated sensor data.
11. The computer system of claim 8 , wherein the measure of the effectiveness of the one or more autonomous operation features includes one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features.
12. The computer system of claim 11 , wherein the one or more risk levels associated with operation of the virtual vehicle by the one or more autonomous operation features are determined based at least in part upon a comparison of (i) the one or more test output signals generated by the one or more software routines, (ii) one or more other test output signals generated by one or more other software routines of one or more other autonomous operation features in response to one or more other test input signals, and (iii) observed operating data regarding the one or more other autonomous operation features disposed within a plurality of other vehicles operating outside the plurality of virtual test environments.
13. The computer system of claim 12 , wherein the observed operating data regarding the one or more other autonomous operation features includes data regarding actual losses associated with insurance policies covering the plurality of other vehicles having the one or more other autonomous operation features.
14. The computer system of claim 8 , wherein each of the plurality of virtual test environments is based at least in part upon observed data regarding actual environments recorded by one or more sensors communicatively connected to a plurality of vehicles operating outside the plurality of virtual test environments.
15. A tangible, non-transitory computer-readable medium storing executable instructions for testing the effectiveness of one or more autonomous operation features for controlling vehicles by simulating operation of the one or more autonomous operation features in a virtual vehicle in a plurality of virtual test environments associated with a plurality of environmental conditions that, when executed by at least one processor of a computer system, cause the computer system to:
receive a set of computer-readable instructions for implementing the one or more autonomous operation features, wherein the set of computer-readable instructions comprises one or more software routines configured to receive one or more input signals from at least one sensor and generate one or more output signals for controlling a vehicle;
cause a sensor input simulator to receive the plurality of virtual test environments;
for each virtual test environment of the plurality of virtual test environments, automatically testing the one or more autonomous operation features without input from a user by:
cause the sensor input simulator to generate a fully virtual vehicle simulation including the virtual vehicle within the virtual test environment;
execute the one or more software routines to respond to test input signals for the virtual vehicle within the fully virtual vehicle simulation;
receive a plurality of test input signals for the virtual test environment from the sensor input simulator, wherein the test input signals simulate the one or more signals from at least one sensor corresponding to a test scenario associated with the virtual test environment;
generate one or more test output signals for the virtual vehicle using the one or more software routines in response to the received test input signals for the virtual test environment;
cause the sensor input simulator to predict one or more responses of the virtual vehicle in the virtual test environment to the one or more test output signals;
determine a measure of the effectiveness of the one or more autonomous operation features based at least in part upon the one or more predicted responses of the virtual vehicle to the one or more test output signals associated with the plurality of virtual test environments; and
adjust a risk profile associated with a vehicle having the one or more autonomous operation features based upon the determined measure of effectiveness by updating the risk profile to include one or more risk levels associated with autonomous operation of the vehicle by the one or more autonomous operation features.
16. The tangible, non-transitory computer-readable medium of claim 15 , wherein the executable instructions that cause the computer system to cause the sensor input simulator to generate the fully virtual vehicle simulation further cause the sensor input simulator to access the one or more test input signals from a database containing a plurality of test signals.
17. The tangible, non-transitory computer-readable medium of claim 15 , wherein the executable instructions that cause the sensor input simulator to generate the fully virtual vehicle simulation further cause the sensor input simulator to:
generate a simulation of the virtual vehicle in the virtual test environment;
determine simulated sensor data associated with the virtual vehicle in the virtual test environment; and
determine the one or more test input signals based at least in part upon the simulated sensor data.
18. The tangible, non-transitory computer-readable medium of claim 15 , wherein the measure of the effectiveness of the one or more autonomous operation features includes one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features.
19. The tangible, non-transitory computer-readable medium of claim 18 , wherein the one or more risk levels associated with operation of the virtual vehicle by the one or more autonomous operation features are determined based at least in part upon a comparison of (i) the one or more test output signals generated by the one or more software routines, (ii) one or more other test output signals generated by one or more other software routines of one or more other autonomous operation features in response to one or more other test input signals, and (iii) observed operating data regarding the one or more other autonomous operation features disposed within a plurality of other vehicles operating outside the plurality of virtual test environments.
20. The tangible, non-transitory computer-readable medium of claim 19 , wherein the observed operating data regarding the one or more other autonomous operation features includes data regarding actual losses associated with insurance policies covering the plurality of other vehicles having the one or more other autonomous operation features.