Non-determinism testing for software for controlling autonomous vehicles
The disclosure relate to detecting differences indicative of non-determinism in software used to control autonomous vehicles. For instance, a simulation may be run using the software at least two times. The output of the simulations may be tracked. The output of the simulations includes messages may be generated by a decision making module of the software that affect vehicle behavior. A pair of messages generated from each of the at least two times the simulation is run may be compared. Based on the comparison, a difference indicative of non-determinism in the software may be detected.
1. A method comprising:
selecting, by one or more processors, a portion of log data stored in one or more memory devices that was generated by one or more sensors of a perception system of a vehicle;
retrieving, by the one or more processors, the portion of log data from the one or more memory devices;
running, by the one or more processors, a first simulation using software and the retrieved portion of log data;
tracking, by the one or more processors, messages generated by the software during the first simulation including first messages that are related to non-determinism in the software and second messages that are not related to non-determinism in the software;
running, by the one or more processors, a second simulation using the software and the retrieved portion of log data;
tracking, by the one or more processors, messages generated by the software during the second simulation including third messages that are related to non-determinism in the software and fourth messages that are not related to non-determinism in the software;
comparing, by the one or more processors, one or more fields of the first messages that were generated by the software during the first simulation with one or more fields of the third messages that were generated by the software during the second simulation; and
based on a result of the comparing, detecting, by the one or more processors, a difference between the one or more fields of the first messages that were generated by the software during the first simulation and the one or more fields of the third messages that were generated by the software during the second simulation that is indicative of non-determinism in the software.
2. The method of claim 1 , further comprising, prior to comparing the fields of the first messages with the one or more fields of the third messages, filtering the second messages and the fourth messages to remove fields unrelated to non-determinism such that only corresponding fields related to non-determinism are compared when comparing the one or more fields of the first messages with the one or more fields of the third messages.
3. The method of claim 1 , wherein the difference indicative of non-determinism is a first difference in time between fields of corresponding first messages of the first simulation and the third messages of the second simulation.
4. The method of claim 1 , wherein comparing the one or more fields of the first messages with the one or more fields of the third messages is based on a tolerance for acceptable differences between corresponding fields of the first messages and the third messages.
5. The method of claim 1 , further comprising, identifying a software module of the software that caused the software to behave non-deterministically based on the difference.
6. The method of claim 1 , further comprising, determining based on the difference a percentage of plurality of simulations for which the software behaves non-deterministically.
7. The method of claim 1 , further comprising, flagging the first and second messages for review.
8. The method of claim 1 , wherein the software is a latest version of the software, and the method further comprising, aggregating differences identified from prior versions of the software over time in order to detect anomalies in the latest version of the software.
9. The method of claim 8 , wherein a detected anomaly indicates an increase in non-determinism from the latest version of the software over the prior versions of the software.
10. The method of claim 8 , wherein a detected anomaly indicates a decrease in non-determinism from the latest version of the software over the prior versions of the software.
11. The method of claim 1 , wherein the messages are generated by a decision making module of the software, which is a planning system configured to generate trajectories for the vehicle to follow.
12. The method of claim 1 , wherein the messages are generated by the perception system, which is configured to detect and identify objects in an environment of the vehicle using sensor data generated by the one or more sensors.
13. The method of claim 1 , wherein the messages are generated by a decision making module of the software, which is a behavior prediction system configured to predict future behaviors of objects detected in an environment of the vehicle from sensor data generated by the one or more sensors.
14. The method of claim 1 , wherein the messages are generated by a decision making module of the software, which is a construction zone detection system configured to detect construction zones from sensor data generated by the one or more sensors.
15. The method of claim 1 , wherein the messages are generated by a decision making module of the software, which is an emergency vehicle detection system configured to detect emergency vehicles from sensor data generated by the one or more sensors.
16. The method of claim 1 , wherein the messages are generated by a decision making module of the software, which is a control system configured to control movement of the vehicle in order to follow a trajectory.
17. The method of claim 1 , wherein the portion of log data corresponds to a period of time during which the log data was generated by the one or more sensors.
18. The method of claim 1 , further comprising:
identifying, by the one or more processors, a cause of the non-determinism based on the result of the comparing.
19. The method of claim 18 , wherein the cause is a planner software module or a perception software module of the software.
20. A system comprising:
one or more processors configured to:
select a portion of log data stored in one or more memory devices that was generated by one or more sensors of a perception system of a vehicle;
retrieve the portion of log data from the one or more memory devices;
run a first simulation using software and the retrieved portion of log data;
track messages generated by the software during the first simulation including first messages that are related to non-determinism in the software and second messages that are not related to non-determinism in the software;
run a second simulation using the software and the retrieved portion of log data;
track messages generated by the software during the second simulation including third messages that are related to non-determinism in the software and fourth messages that are not related to non-determinism in the software;
compare one or more fields of the first messages that were generated by the software during the first simulation with one or more fields of the third messages that were generated by the software during the second simulation; and
based on a result of the comparison, detect a difference between the one or more fields of the first messages that were generated by the software during the first simulation and the one or more fields of the third messages that were generated by the software during the second simulation that is indicative of non-determinism in the software.
21. The system of claim 20 , wherein the one or more processors are further configured to, prior to comparing the one or more fields of the first messages with the one or more fields of the third messages, filter the second messages and the fourth messages to remove fields unrelated to non-determinism such that only corresponding fields related to non-determinism are compared when comparing the one or more fields of the first messages with the one or more fields of the third messages.
22. The system of claim 20 , wherein the difference indicative of non-determinism is a first difference in time between fields of corresponding first messages of the first simulation and the third messages of the second simulation.
23. The system of claim 20 , wherein comparing the one or more fields of the first messages with the one or more fields of the third messages is based on a tolerance for acceptable differences between corresponding fields of the first messages and the third messages.
24. The system of claim 20 , wherein the portion of log data corresponds to a period of time during which the log data was generated by the one or more sensors.