Conversion of driving logs to simulations for autonomous vehicle testing and validation

1 . A system comprising:

one or more processors; and

non-transitory memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform actions including:

accessing a driving log associated with a first autonomous vehicle in an environment, the driving log comprising data relating to the environment and at least one object within the environment;

generating, by a simulator of the system, a driving simulation including a simulated environment representing the environment and at least one simulated object representing the at least one object based at least in part on the driving log;

determining at least one confidence score for the driving simulation, wherein the at least one confidence score indicates a quality of the simulator, the at least one confidence score associated with the at least one simulated object;

including, based at least in part on determining that the at least one confidence score satisfies a condition for inclusion of the driving simulation in a dataset to validate a machine-learned model of a second autonomous vehicle, the driving simulation in the dataset to validate the machine-learned model of the second autonomous vehicle; and

validating the machine-learned model of the second autonomous vehicle based at least in part on the dataset.

2 . The system of claim 1 , wherein the at least one confidence score is based at least in part on a lack of a model for the at least one simulated object to represent the at least one object.

3 . The system of claim 1 , wherein the at least one confidence score is based at least in part on an absence of data for the at least one object in the driving log to fully render the at least one simulated object.

4 . The system of claim 3 , wherein the absence of the data is an absence of location data corresponding to a location of the at least one object or a portion of the at least one object.

5 . The system of claim 1 , wherein the at least one confidence score is based on at least one of:

a distance of the at least one object from the first autonomous vehicle based at least in part on the driving log;

a reliance on the at least one simulated object for determining an action by a planning component of the first autonomous vehicle based at least in part on the driving log; or

a purpose of the driving simulation.

6 . A system comprising:

one or more processors; and

non-transitory memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform actions including:

receiving a driving log associated with a vehicle;

determining, from the driving log, log data that represents a driving scenario;

determining, based at least in part on the log data, at least one feature of the driving scenario;

generating, by a simulation system, a driving simulation of the driving scenario based at least in part on the log data;

determining a value for the driving simulation that indicates an accuracy of the simulation system, the value associated with a simulated representation of the at least one feature of the driving scenario in the driving simulation; and

storing, based at least in part on determining that the value satisfies a condition for inclusion of the driving simulation in a database to validate a machine-learned model of a second vehicle, the driving simulation in the database.

7 . The system of claim 6 , wherein the instructions, when executed by the one or more processors, cause the system to perform actions including:

generating the driving simulation of the driving scenario based at least in part on at least one of stored map data or stored road network data;

determining that the driving simulation comprises the at least one feature based at least in part on the stored map data or the stored road network data; and

determining the value based at least in part on determining that the driving simulation comprises the at least one feature.

8 . The system of claim 6 , wherein the instructions, when executed by the one or more processors, cause the system to perform actions including:

determining, based at least in part on the log data, that the system has limited or no ability to represent the at least one feature of the driving scenario; and

determining the value based at least in part on determining that the system has limited or no ability to represent the at least one feature.

9 . The system of claim 8 , wherein the system has limited or no ability to represent the at least one feature due to a lack of data corresponding to the at least one feature in the log data.

10 . The system of claim 6 , wherein the value is based at least in part on one or more of:

a distance of the at least one feature from the vehicle;

a reliance on the at least one feature by the vehicle to perform an action; or

a purpose of the driving simulation.

11 . The system of claim 6 , wherein the instructions, when executed by the one or more processors, cause the system to perform actions including:

accessing a database comprising a plurality of simulated features;

determining a simulated feature of the plurality of simulated features for representing the at least one feature in the driving simulation based at least in part on the log data;

determining a difference between the at least one feature and the simulated feature; and

determining the value based at least in part on the difference.

12 . The system of claim 11 , wherein the difference comprises at least one of:

a shape difference;

a size difference;

a difference in material;

a textural difference; or

a color difference.

13 . The system of claim 6 , wherein the value comprises an overall value and the at least one feature comprises a first feature and a second feature, and wherein the instructions, when executed by the one or more processors, cause the system to perform actions including:

determining a first value that indicates an ability of the system to represent the first feature in the driving simulation based at least in part on the log data;

determining a second value that indicates an ability of the system to represent the second feature in the driving simulation based at least in part on the log data; and

determining the overall value based at least in part on the first value and the second value.

14 . The system of claim 6 , wherein the instructions, when executed by the one or more processors, cause the system to perform actions including:

executing the driving simulation;

determining, based on executing the driving simulation, data associated with the driving simulation; and

determining the value for the driving simulation based at least in part on at least one of executing the driving simulation or the data associated with the driving simulation.

15 . One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform actions including:

receiving a driving log associated with a vehicle;

determining, from the driving log, log data that represents a driving scenario;

determining, based at least in part on the log data, at least one feature of the driving scenario;

generating, by a simulation system, a driving simulation of the driving scenario based at least in part on the log data;

determining a value for the driving simulation that indicates an accuracy of the simulation system, the value associated with a simulated representation of the at least one feature of the driving scenario in the driving simulation; and

storing, based at least in part on determining that the value satisfies a condition for inclusion of the driving simulation in a database to validate a machine-learned model of a second vehicle, the driving simulation in the database.

16 . The computer-readable media of claim 15 , wherein the instructions, when executed, cause the processor to perform actions including:

determining that the value is less than a threshold value; and

performing a further action based at least in part on determining that the value is less than the threshold value.

17 . The computer-readable media of claim 16 , wherein the further action comprises at least one of:

marking the driving simulation to indicate that the value is less than the threshold value;

generating a request for user input;

generating a new driving simulation based at least in part on the driving log, the new driving simulation including a new simulated feature corresponding to the at least one feature, wherein the new simulated feature is different to a simulated feature corresponding to the at least one feature found in the driving simulation.

18 . The computer-readable media of claim 15 , wherein the instructions, when executed, cause the processor to perform actions including presenting the data associated with the driving simulation and the value to a user in an ordered list comprising data associated with a plurality of driving simulations and associated values.

19 . The computer-readable media of claim 15 , wherein the instructions, when executed, cause the processor to perform actions including:

determining that the processor has limited or no ability to represent the at least one feature of the driving scenario; and

determining the value based at least in part on determining that the processor has limited or no ability to represent the at least one feature.

20 . The computer-readable media of claim 15 , wherein the instructions, when executed by the one or more processors, cause the processor to perform actions including:

executing the driving simulation;

determining, based on executing the driving simulation, data associated with the driving simulation; and

determining the value for the driving simulation based at least in part on at least one of executing the driving simulation or the data associated with the driving simulation.