Risk determination for autonomous vehicle simulations
System, methods, and computer-readable media for determining metrics associated with simulations of proximity events within autonomous vehicle simulations. In some examples, a simulation may be created using driving log data from an autonomous vehicle. Portions of the simulation containing proximity events may be analyzed. Intent times for both the autonomous vehicle and agent vehicles within the simulation may be determined, and the deceleration required for each vehicle to avoid the proximity event may be calculated. These decelerations may be used to determine metrics for the likeliness of the proximity event and the avoidability of the proximity event by the autonomous vehicle. These metrics may be used to prioritize development tasks, assess performance of the simulation, and assess performance of vehicle control systems.
1 . A system comprising:
one or more processors; and
non-transitory memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform actions including:
accessing a driving log comprising real perception data of a driving scenario within a physical environment;
generating a simulation based at least in part on the driving scenario in the driving log, the simulation comprising a simulated autonomous vehicle in a simulated environment, the simulated autonomous vehicle being controlled in the simulation by a control code portion, and at least one simulated agent vehicle controlled at least in part by data from the driving log in the simulated environment;
identifying a portion of the simulation wherein the simulated autonomous vehicle has a proximity event with a simulated agent vehicle in a proximity zone; and
determining, for the portion of the simulation:
a first deceleration that would be required by the simulated agent vehicle to avoid the proximity event;
a first value for a first metric representative of the first deceleration;
a second deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event;
a second value for a second metric representative of a reciprocal of the second deceleration; and
a classification of the proximity event as an avoidable or an unavoidable event based on the first value and the second value.
2 . The system of claim 1 , wherein the first deceleration is determined at a time that the simulated autonomous vehicle is on course to occupy the proximity zone; and the second deceleration is determined at a time that the simulated agent vehicle is on course to occupy the proximity zone.
3 . The system of claim 2 , wherein the time that the simulated autonomous vehicle is on course to occupy the proximity zone, and the time that the simulated agent vehicle is on course to occupy the proximity zone are each determined using at least one of a respective:
longitudinal intent time; and
lateral intent time.
4 . The system of claim 3 , wherein the longitudinal intent time comprises a first time at which an entity's speed indicates intent to occupy the proximity zone, calculated based on a predetermined rate of deceleration; and wherein the lateral intent time comprises a first time at which it is clear the entity's path will lead to the proximity zone.
5 . A method comprising:
accessing a driving log comprising real perception data determined for a physical environment;
generating a simulation based at least in part on the driving log, the simulation comprising a simulated autonomous vehicle in a simulated environment and a plurality of simulated agent vehicles in the simulated environment;
identifying a portion of the simulation wherein the simulated autonomous vehicle has a proximity event with at least one simulated agent vehicle;
determining, for the portion of the simulation:
a first metric representative of a likeliness of the proximity event occurring; and
a second metric representative of a reciprocal of a deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event; and
classifying, for the portion of the simulation and based on at least the second metric, the proximity event as an avoidable or an unavoidable event.
6 . The method of claim 5 , wherein:
the first metric comprises a measure of a first deceleration required by the simulated agent vehicle to avoid the proximity event at a time that the simulated autonomous vehicle is on course to occupy a proximity zone where the proximity event occurs;
and wherein:
the second metric comprises a measure of an inverse of a second deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event at a time that the simulated agent vehicle is on course to occupy the proximity zone where the proximity event occurs.
7 . The method of claim 6 , wherein the time that the simulated autonomous vehicle is on course to occupy the proximity zone, and the time that the simulated agent vehicle is on course to occupy the proximity zone are each determined using at least one of a respective:
longitudinal intent time; and
lateral intent time.
8 . The method of claim 7 , wherein the longitudinal intent time comprises the first time at which an entity's speed indicates intent to occupy the proximity zone, calculated based on a predetermined rate of deceleration; and wherein the lateral intent time comprises the first time at which it is clear the entity's path will lead to the proximity zone.
9 . The method of claim 8 , wherein the predetermined rate of deceleration is one of several predetermined rates of deceleration each corresponding to a respective category of the agent.
10 . The method of claim 9 , wherein the category comprises one of:
Car;
Large vehicle;
Motorcycle;
Aircraft;
Vehicle with towed load;
Bicycle;
Animal; and
Pedestrian.
11 . The method of claim 6 , wherein the first metric is determined by generating one or more further simulations in which the simulated agent decelerates at different times in order to determine a latest time at which the proximity event is avoided.
12 . The method of claim 6 , wherein the second metric is determined by generating one or more further simulations in which the simulated autonomous vehicle decelerates at different times in order to determine a latest time at which the proximity event is avoided.
13 . The method of claim 5 , further comprising training a machine learning model using at least one data item from a driving log in a case that the proximity event is classified as an avoidable proximity event.
14 . The method of claim 5 , further comprising ranking a plurality of proximity events according to at least one of respective first or second metrics.
15 . The method of claim 13 , further comprising selecting at least one proximity event of the plurality of proximity events for use in evaluating at least one machine learning model, based at least in part on at least one of:
the respective first metric;
the respective second metric
a position of the at least one proximity events within the ranking of the plurality of proximity events; and
the classification of the proximity event as avoidable or unavoidable.
16 . The method of claim 5 , further comprising determining a trend across the plurality of simulations according to respective first and second metrics.
17 . One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
accessing a driving log comprising real perception data determined for a physical environment;
generating a simulation based at least in part on the driving log, the simulation comprising a simulated autonomous vehicle in a simulated environment and a plurality of simulated agent vehicles in the simulated environment;
identifying a portion of the simulation wherein the simulated autonomous vehicle has a proximity event with at least one simulated agent vehicle;
determining, for the portion of the simulation:
a first metric representative of a likeliness of the proximity event occurring; and
a second metric representative of a reciprocal of a deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event; and
classifying, for the portion of the simulation and based on at least the second metric, the proximity event as an avoidable or an unavoidable event.
18 . The non-transitory computer-readable media of claim 17 , wherein:
the first metric comprises a measure of a first deceleration required by the simulated agent vehicle to avoid the proximity event at a time that the simulated autonomous vehicle is on course to occupy a proximity zone where the proximity event occurs;
and wherein:
the second metric comprises a measure of an inverse of a second deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event at a time that the simulated agent vehicle is on course to occupy the proximity zone where the proximity event occurs.
19 . The non-transitory computer-readable media of claim 18 , wherein the time that the simulated autonomous vehicle is on course to occupy the proximity zone, and the time that the simulated agent vehicle is on course to occupy the proximity zone are each determined using at least one of a respective:
longitudinal intent time; and
lateral intent time.
20 . The non-transitory computer-readable media of claim 19 , wherein the longitudinal intent time comprises the first time at which an entity's speed indicates intent to occupy the proximity zone, calculated based on a predetermined rate of deceleration; and wherein the lateral intent time comprises the first time at which it is clear the entity's path will lead to the proximity zone.