Parameter space optimization
Techniques for analyzing a parameter space are discussed. Techniques may include receiving policy data for evaluating a vehicle controller. The techniques may further include determining, using a Bayesian optimization and based at least in part on the vehicle controller, parameter sets associated with adverse events. The adverse events may be associated with a violation of the policy data. The techniques may associate, based on exposure data, parameter bounds of the adverse events and probabilities of the adverse events in a driving environment. A safety metric may be determined based on the Bayesian optimization. The techniques may also include weighting an impact of an adverse event based on the safety metric.
1 . A method comprising:
receiving policy data for evaluating a vehicle controller;
determining, using a Bayesian optimization and based at least in part on the vehicle controller, parameter sets associated with adverse events, wherein the adverse events are associated with a violation of the policy data, wherein parameters in the parameter sets represent conditions of the vehicle controller and an object in a simulated environment, determined using the Bayesian optimization and based on the vehicle controller, resulting in an increased likelihood of a simulation resulting in the adverse events between the vehicle controller and the object;
associating, based on exposure data, bounds of the parameters of the parameters sets of the adverse events and probabilities of the adverse events in a driving environment, wherein the bounds of the parameters in the parameter sets include a minimum value and a maximum value of a distance, a velocity, or an acceleration;
determining a ratio between first results associated with first parameter value ranges indicating an adverse event occurred and second results associated with second parameter value ranges indicating a lack of the adverse event;
determining a safety metric based on the Bayesian optimization and the ratio; and
assigning a safety metric value to the adverse event.
2 . The method of claim 1 , wherein determining parameter sets associated with adverse events includes:
determining results of previous Bayesian optimizations; and
determining, based on the results of previous Bayesian optimizations, low frequency adverse events using a first Bayesian acquisition function.
3 . The method of claim 1 , wherein the parameters are first parameters, and wherein second parameters in the parameter sets comprise one or more of:
a number of objects in an environment;
a type of an object in an environment;
a type of intersection;
a vehicle speed;
a distance from a vehicle represented by the vehicle controller to an additional object; and
a speed of the additional object.
4 . The method of claim 1 , wherein the exposure data includes data captured by at least one vehicle during driving activities.
5 . The method of claim 1 , further comprising:
determining whether the safety metric exceeds a threshold value for use in an autonomous vehicle.
6 . The method of claim 5 , further comprising:
validating the Bayesian optimization for use in an autonomous vehicle controller responsive to determining that the safety metric exceeds the threshold value.
7 . The method of claim 1 , wherein associating the bounds of the parameters of the parameters sets of the adverse events and probabilities of the adverse events is performed using a Gaussian mixture model.
8 . The method of claim 1 , wherein the Bayesian optimization uses a first acquisition function, the method further comprising:
iteratively evaluating the vehicle controller using the first acquisition function; and
based at least in part on a result of the first acquisition function, executing the vehicle controller using a second acquisition function;
wherein the first acquisition function is associated with exploring a parameter space and the second acquisition function is associated with optimizing simulations within the parameter space that are associated with low frequency adverse events.
9 . The method of claim 1 , wherein an adverse event of the adverse events includes an event in which executing the vehicle controller results in a violation of the policy data.
10 . The method of claim 1 , wherein the policy data comprises one or more of:
a requirement to maintain a minimum distance between a vehicle representing the vehicle controller and another object;
a requirement to stay below a maximum speed;
a requirement to identify all objects in the driving environment; and
a requirement to stay within a driving lane.
11 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving policy data for evaluating a vehicle controller;
determining, using a Bayesian optimization and based at least in part on the vehicle controller, parameter sets associated with adverse events, wherein the adverse events are associated with a violation of the policy data, wherein parameters in the parameter sets represent conditions of the vehicle controller and an object in a simulated environment, determined using the Bayesian optimization and based on the vehicle controller, resulting in an increased likelihood of a simulation resulting in the adverse events between the vehicle controller and the object;
associating, based on exposure data, bounds of the parameters in the parameter sets of the adverse events and probabilities of the adverse events in a driving environment, wherein the bounds of parameters in the parameter sets include a minimum value and a maximum value of a distance, a velocity, or an acceleration;
determining a ratio between first results associated with first parameter value ranges indicating an adverse event occurred and second results associated with second parameter value ranges indicating a lack of the adverse event;
determining, based at least in part on the ratio, a safety metric;
assigning a safety metric value to the adverse event; and
determining whether the safety metric exceeds a threshold value for use in an autonomous vehicle.
12 . The system of claim 11 , wherein the exposure data includes data captured by at least one vehicle during driving activities.
13 . The system of claim 11 , wherein associating the bounds of the parameters in the parameter sets of the adverse events and probabilities of the adverse events is performed using a Gaussian mixture model.
14 . The system of claim 11 , wherein the Bayesian optimization uses a first acquisition function, the operations further comprising:
iteratively evaluating the vehicle controller using the first acquisition function; and
based at least in part on a result of the first acquisition function, executing the vehicle controller using a second acquisition function;
wherein the first acquisition function is associated with exploring a parameter space and the second acquisition function is associated with optimizing simulations within the parameter space that are associated with low frequency adverse events.
15 . The system of claim 11 , wherein the adverse event includes an event in which executing the vehicle controller results in a violation of the policy data.
16 . The system of claim 11 , wherein the policy data comprises one or more of:
a requirement to maintain a minimum distance between the autonomous vehicle and another object;
a requirement to stay below a maximum speed;
a requirement to identify all objects in the driving environment; and
a requirement to stay within a driving lane.
17 . One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
receiving policy data for evaluating a vehicle controller;
determining, using a Bayesian optimization and based at least in part on the vehicle controller, parameter sets associated with adverse events, wherein the adverse events are associated with a violation of the policy data, wherein parameters in the parameter sets represent conditions of the vehicle controller and an object in a simulated environment, determined using the Bayesian optimization and based on the vehicle controller, resulting in an increased likelihood of a simulation resulting in the adverse events between the vehicle controller and the object;
associating, based on exposure data, bounds of the parameters in the parameter sets of the adverse events and probabilities of the adverse events in a driving environment, wherein the bounds of the parameters in the parameter sets include a minimum value and a maximum value of a distance, a velocity, and an acceleration;
determining a ratio between first results associated with first parameter value ranges indicating an adverse event occurred and second results associated with second parameter value ranges indicating a lack of the adverse event;
determining a safety metric based on the Bayesian optimization and the ratio; and
assigning a safety metric value to the adverse event.
18 . The one or more non-transitory computer-readable media of claim 17 , wherein determining parameter sets associated with adverse events includes:
determining results of previous Bayesian optimizations; and
determining, based on the results of previous Bayesian optimizations, low frequency adverse events using a first Bayesian acquisition function.
19 . The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:
validating the Bayesian optimization for use in an autonomous vehicle controller responsive to determining that the safety metric exceeds a threshold value.
20 . The one or more non-transitory computer-readable media of claim 17 , wherein associating the bounds of the parameters in the parameter sets of the adverse events and probabilities of the adverse events is performed using a Gaussian mixture model.