Methods and Systems for Comparing Resultant Status of Autonomous Vehicle Simulations
Systems providing a comparison of results of simulations of operation of a simulated autonomous vehicle may include a processor to: perform a first simulation of operation of a simulated autonomous vehicle based on first autonomous vehicle control code, receive second autonomous vehicle control code that includes a second version of software code associated with controlling operations of the simulated autonomous vehicle, the second version including a modification of a first version of software code associated with controlling operations of the simulated autonomous vehicle, perform a second simulation of operation of the simulated autonomous vehicle based on the second autonomous vehicle control code, and display an indication that second values of one or more metrics that resulted from the second simulation are different from one or more first values of the one or more metrics that resulted from the first simulation. Methods, computer program products, and autonomous vehicles are also disclosed.
1 . A system comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive first autonomous vehicle control code, wherein the first autonomous vehicle control code comprises a first version of software code associated with controlling operations of a simulated autonomous vehicle;
perform a first simulation of operation of the simulated autonomous vehicle based on the first autonomous vehicle control code, wherein the first simulation of operation is associated with one or more metrics that measure performance of the simulated autonomous vehicle;
receive second autonomous vehicle control code, wherein the second autonomous vehicle control code comprises a second version of software code associated with controlling operations of the simulated autonomous vehicle, wherein the second version of software code associated with controlling operations of the simulated autonomous vehicle comprises a modification of the first version of software code associated with controlling operations of the simulated autonomous vehicle;
perform a second simulation of operation of the simulated autonomous vehicle based on the second autonomous vehicle control code, wherein the second simulation of operation is associated with the one or more metrics that measure performance of the simulated autonomous vehicle; and
display an indication, in a user interface, that one or more second values of the one or more metrics that resulted from the second simulation of operation are different from the one or more first values of the one or more metrics that resulted from the first simulation of operation.
2 . The system of claim 1 , wherein the at least one processor is further configured to:
display the one or more first values of the one or more metrics that resulted from the first simulation of operation in the user interface; and
display one or more second values of the one or more metrics that resulted from the second simulation of operation in the user interface.
3 . The system of claim 1 , wherein the at least one processor is further configured to:
determine whether the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy one or more thresholds; and
display an indication, in the user interface, that the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy the one or more thresholds based on determining that the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy the one or more thresholds.
4 . The system of claim 1 , wherein the at least one processor is further configured to:
display one or more maximum values of the one or more metrics that resulted from the second simulation of operation in the user interface.
5 . The system of claim 1 , wherein the one or more metrics that measure performance of the simulated autonomous vehicle comprise:
a metric associated with a minimum distance between the simulated autonomous vehicle and an actor experienced during a simulation of operation;
a metric associated with a number of halt events experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of longitudinal deceleration experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of longitudinal negative jerk experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of lateral acceleration experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of lateral jerk experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with a number of times the simulated autonomous vehicle planned to stop;
a metric associated with a number of times the simulated autonomous vehicle planned to change a trajectory; or
any combination thereof.
6 . The system of claim 1 , wherein the at least one processor is further configured to:
display an amount by which the one or more second values of the one or more metrics are different than the one or more first values of the one or more metrics.
7 . The system of claim 1 , wherein the at least one processor is further configured to:
provide operational code to an autonomous vehicle for operating the autonomous vehicle in a real environment, wherein the operational code is based on the second autonomous vehicle control code; and
operate the autonomous vehicle in the real environment based on the operational code.
8 . The system of claim 1 , wherein the at least one processor is further configured to:
perform a plurality of additional simulations of operation of the simulated autonomous vehicle, wherein each additional simulation of operation is performed based on additional autonomous vehicle control code, wherein the additional autonomous vehicle control code comprises an additional version of software code associated with controlling operations of the simulated autonomous vehicle;
display, in the user interface, a trend graph of values of the one or more metrics that resulted from the plurality of additional simulations of operation of the simulated autonomous vehicle; and
display an indication of the one or more second values of the one or more metrics that resulted from the second autonomous vehicle control code in comparison to the trend graph.
9 . A computer program product comprising at least one non-transitory computer-readable medium comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to:
receive first autonomous vehicle control code, wherein the first autonomous vehicle control code comprises a first version of software code associated with controlling operations of a simulated autonomous vehicle;
perform a first simulation of operation of the simulated autonomous vehicle based on the first autonomous vehicle control code, wherein the first simulation of operation is associated with one or more metrics that measure performance of the simulated autonomous vehicle;
receive second autonomous vehicle control code, wherein the second autonomous vehicle control code comprises a second version of software code associated with controlling operations of the simulated autonomous vehicle, wherein the second version of software code associated with controlling operations of the simulated autonomous vehicle comprises a modification of the first version of software code associated with controlling operations of the simulated autonomous vehicle;
perform a second simulation of operation of the simulated autonomous vehicle based on the second autonomous vehicle control code, wherein the second simulation of operation is associated with the one or more metrics that measure performance of the simulated autonomous vehicle; and
display an indication, in a user interface, that one or more second values of the one or more metrics that resulted from the second simulation of operation are different from the one or more first values of the one or more metrics that resulted from the first simulation of operation.
10 . The computer program product of claim 9 , wherein the one or more instructions further cause the at least one processor to:
display the one or more first values of the one or more metrics that resulted from the first simulation of operation in the user interface; and
display one or more second values of the one or more metrics that resulted from the second simulation of operation in the user interface.
11 . The computer program product of claim 9 , wherein the one or more instructions further cause the at least one processor to:
determine whether the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy one or more thresholds; and
display an indication, in the user interface, that the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy the one or more thresholds based on determining that the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy the one or more thresholds.
12 . The computer program product of claim 9 , wherein the one or more instructions further cause the at least one processor to:
display one or more maximum values of the one or more metrics that resulted from the second simulation of operation in the user interface.
13 . The computer program product of claim 9 , wherein the one or more metrics that measure performance of the simulated autonomous vehicle comprise:
a metric associated with a minimum distance between the simulated autonomous vehicle and an actor experienced during a simulation of operation;
a metric associated with a number of halt events experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of longitudinal deceleration experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of longitudinal negative jerk experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of lateral acceleration experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of lateral jerk experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with a number of times the simulated autonomous vehicle planned to stop;
a metric associated with a number of times the simulated autonomous vehicle planned to change a trajectory; or
any combination thereof.
14 . The computer program product of claim 9 , wherein the one or more instructions further cause the at least one processor to:
provide operational code to an autonomous vehicle for operating the autonomous vehicle in a real environment, wherein the operational code is based on the second autonomous vehicle control code; and
operate the autonomous vehicle in the real environment based on the operational code.
15 . The computer program product of claim 9 , wherein the one or more instructions further cause the at least one processor to:
perform a plurality of additional simulations of operation of the simulated autonomous vehicle, wherein each additional simulation of operation is performed based on an additional version of software code associated with controlling operations of the simulated autonomous vehicle;
display, in the user interface, a trend graph of values of the one or more metrics that resulted from the plurality of additional simulations of operation of the simulated autonomous vehicle; and
display an indication of the one or more second values of the one or more metrics that resulted from the second autonomous vehicle control code in comparison to the trend graph.
16 . A method, comprising:
performing, with at least one processor, a first simulation of operation of a simulated autonomous vehicle based on first autonomous vehicle control code, wherein the first simulation of operation is associated with one or more metrics that measure performance of the simulated autonomous vehicle;
receiving, with the at least one processor, second autonomous vehicle control code, wherein the second autonomous vehicle control code comprises a second version of software code associated with controlling operations of the simulated autonomous vehicle, wherein the second version of software code associated with controlling operations of the simulated autonomous vehicle comprises a modification of the first version of software code associated with controlling operations of the simulated autonomous vehicle;
performing, with the at least one processor, a second simulation of operation of the simulated autonomous vehicle based on the second autonomous vehicle control code, wherein the second simulation of operation is associated with the one or more metrics that measure performance of the simulated autonomous vehicle; and
displaying, with the at least one processor, an indication, in a user interface, that one or more second values of the one or more metrics that resulted from the second simulation of operation are different from the one or more first values of the one or more metrics that resulted from the first simulation of operation.
17 . The method of claim 16 , further comprising:
determining whether the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy one or more thresholds; and
displaying an indication, in the user interface, that the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy the one or more thresholds based on determining that the one or more second values of the one or more metrics that resulted from the second simulation of operation satisfy the one or more thresholds.
18 . The method of claim 16 , wherein the one or more metrics that measure performance of the simulated autonomous vehicle comprise:
a metric associated with a minimum distance between the simulated autonomous vehicle and an actor experienced during a simulation of operation;
a metric associated with a number of halt events experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of longitudinal deceleration experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of longitudinal negative jerk experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of lateral acceleration experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with an amount of lateral jerk experienced by the simulated autonomous vehicle during a simulation of operation;
a metric associated with a number of times the simulated autonomous vehicle planned to stop;
a metric associated with a number of times the simulated autonomous vehicle planned to change a trajectory; or
any combination thereof.
19 . The method of claim 16 , further comprising:
providing operational code to an autonomous vehicle for operating the autonomous vehicle in a real environment, wherein the operational code is based on the second autonomous vehicle control code; and
operating the autonomous vehicle in the real environment based on the operational code.
20 . The method of claim 16 , further comprising:
performing a plurality of additional simulations of operation of the simulated autonomous vehicle before performing the second simulation, wherein each additional simulation of operation is performed based on an additional version of software code associated with controlling operations of the simulated autonomous vehicle;
displaying, in the user interface, a trend graph of values of the one or more metrics that resulted from the plurality of additional simulations of operation of the simulated autonomous vehicle; and
displaying an indication of the one or more second values of the one or more metrics that resulted from the second autonomous vehicle control code in comparison to the trend graph.