Application of mean time between failure (MTBF) models for autonomous vehicles
To receive authority certification for mass deployment of autonomous vehicles (AVs), manufacturers need to justify that their AVs operate safer than human drivers. This in turn creates the need to estimate and model the collision rate (failure rate) of an AV taking all possible errors and driving situations into account. In other words, there is the strong demand for comprehensive Mean Time between Failure (MTBF) models for AVs. The disclosure describes such a generic and scalable model that creates a link between errors in the perception system to vehicle-level failures (collisions). Using this model, requirements for the perception quality may then be derived based on the desired vehicle-level MTBF, or vice versa, to obtain an MTBF value given a certain mission profile and perception quality.
1 . A computing device, comprising:
a memory configured to store computer-readable instructions; and
one or more processors configured to execute the computer readable instructions stored in the memory to:
identify perception errors as a result of execution of a perception algorithm implemented via a vehicle comprising an autonomous or a semi-autonomous vehicle, which operates in accordance with a driving policy;
provide a subset of the perception errors that (i) satisfy a first predetermined set of constraints that define the perception errors as safety-relevant to the vehicle due to the perception errors resulting in a change in a safety decision of a planning system of the vehicle, and (ii) further satisfy a second predetermined set of constraints that define the perception errors as being severe due to the perception errors resulting in a collision that exceeds a predefined severity threshold;
correlate each perception error from among the subset of perception errors to a respective set of predetermined driving conditions to determine a perception error rate per each respective set of predetermined driving conditions;
access data from a driving dataset to compute a set of situational probabilities, each respective one of the situational probabilities corresponding to a probability of the vehicle driving in conditions that match each of the respective set of predetermined driving conditions;
compute a vehicle-level failure rate of the vehicle due to the perception errors based upon the set of situational probabilities and the perception error rate per each respective set of predetermined driving conditions;
determine whether the vehicle is compliant with a predetermined safety goal based upon the vehicle-level failure rate; and
when the vehicle is not compliant with the predetermined safety goal, modify a manner in which the perception algorithm is executed for one or more of the respective set of predetermined driving conditions until the vehicle is compliant with the predetermined safety goal.
2 . The computing device of claim 1 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to:
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify, via one or more processors, one or more parameters identified with the perception algorithm based upon the set of situational probabilities; and
repeat the computing of the vehicle-level failure rate and the modifying of the one or more parameters identified with the perception algorithm until the vehicle-level failure rate of the vehicle meets the predetermined safety goal.
3 . The computing device of claim 1 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to:
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify, via one or more processors, one or more parameters of the driving policy based upon the set of situational probabilities; and
repeat the computing of the vehicle-level failure rate and the modifying of the one or more parameters identified with the driving policy until the vehicle-level failure rate of the vehicle meets the predetermined safety goal,
wherein the one or more parameters identified with the driving policy that are modified include at least one of (i) a minimum longitudinal distance between the vehicle and other vehicles, and (ii) a minimum lateral distance between the vehicle and other vehicles while the vehicle is driving.
4 . The computing device of claim 1 , wherein the vehicle-level failure rate represents an inverse of a mean time between failure (MTBF) of the vehicle due to the subset of perception errors resulting in a collision.
5 . The computing device of claim 1 ,
wherein the predefined severity threshold comprises a S2 or S3 severity according to the International Organization for Standardization (ISO) 26262 Standard.
6 . The computing device of claim 1 , wherein each respective set of predetermined driving conditions represents a combination of (i) a predetermined speed range, and (ii) a predetermined set of conditions that define a potentially dangerous traffic situation.
7 . The computing device of claim 6 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to:
compute the vehicle-level failure rate based upon a combination of the set of situational probabilities that are computed for each respective set of predetermined driving conditions.
8 . The computing device of claim 1 , wherein the second predetermined set of constraints yield only safety-relevant perception errors that may result in a collision of a predetermined level of severity that is based upon a delta-velocity at an expected collision time exceeding a predetermined velocity.
9 . The computing device of claim 1 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to identify the perception errors by accessing sensor data from a database, and
when the sensor data stored in the sensor database is less than a threshold size, to identify the perception errors using data augmentation.
10 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by processing circuitry of a computing device, cause the computing device to:
identify perception errors as a result of execution of a perception algorithm implemented via a vehicle comprising an autonomous or a semi-autonomous vehicle, which operates in accordance with a driving policy;
provide a subset of the perception errors that (i) satisfy a first predetermined set of constraints that define the perception errors as safety-relevant to the vehicle due to the perception errors resulting in a change in a safety decision of a planning system of the vehicle, and (ii) further satisfy a second predetermined set of constraints that define the perception errors as being severe due to the perception errors resulting in a collision that exceeds a predefined severity threshold;
correlate each perception error from among the subset of perception errors to a respective set of predetermined driving conditions to determine a perception error rate per each respective set of predetermined driving conditions;
access data from a driving dataset to compute a set of situational probabilities, each respective one of the situational probabilities corresponding to a probability of the vehicle driving in conditions that match each of the respective set of predetermined driving conditions;
compute a vehicle-level failure rate of the vehicle due to the perception errors based upon the set of situational probabilities and the perception error rate per each respective set of predetermined driving conditions;
determine whether the vehicle is compliant with a predetermined safety goal based upon the vehicle-level failure rate; and
when the vehicle is not compliant with the predetermined safety goal, modify a manner in which the perception algorithm is executed for one or more of the respective set of predetermined driving conditions until the vehicle is compliant with the predetermined safety goal.
11 . The non-transitory computer-readable medium of claim 10 , wherein the computer-readable instructions, when executed by the processing circuitry, further cause the computing device to:
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify one or more parameters identified with the perception algorithm based upon the set of situational probabilities; and
repeat the computing of the vehicle-level failure rate and the modifying of the one or more parameters identified with the perception algorithm until the vehicle-level failure rate of the vehicle meets the predetermined safety goal.
12 . The non-transitory computer-readable medium of claim 10 , wherein the computer-readable instructions, when executed by the processing circuitry, further cause the computing device to:
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify one or more parameters of the driving policy based upon the set of situational probabilities; and
repeat the computing of the vehicle-level failure rate and the modifying of the one or more parameters of the driving policy until the vehicle-level failure rate of the vehicle meets the predetermined safety goal.
13 . A computing device, comprising:
a memory configured to store computer-readable instructions; and
one or more processors configured to execute the computer readable instructions stored in the memory to:
identify perception errors as a result of execution of a perception algorithm implemented via a vehicle comprising an autonomous or a semi-autonomous vehicle, which operates in accordance with a driving policy;
provide, via one or more processors, a subset of the perception errors that (i) satisfy a first predetermined set of constraints that define the perception errors as safety-relevant to the vehicle due to the perception errors resulting in a change in a safety decision of a planning system of the vehicle, and (ii) further satisfy a second predetermined set of constraints that define the perception errors as being severe due to the perception errors resulting in a collision that exceeds a predefined severity threshold;
correlate, via one or more processors, each perception error from among the subset of perception errors to a respective set of predetermined driving conditions to determine a perception error rate per each respective set of predetermined driving conditions;
access, via one or more processors, data from a driving dataset to compute a set of situational probabilities, each respective one of the situational probabilities corresponding to a probability of the vehicle driving in conditions that match each of the respective set of predetermined driving conditions;
compute, via one or more processors, a vehicle-level failure rate of the vehicle due to the perception errors based upon the set of situational probabilities and the perception error rate per each respective set of predetermined driving conditions;
determine whether the vehicle is compliant with a predetermined safety goal based upon the vehicle-level failure rate; and
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify, via one or more processors, one or more parameters of the driving policy based upon the set of situational probabilities until the vehicle-level failure rate of the vehicle meets the predetermined safety goal.
14 . The computing device of claim 13 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to:
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify one or more parameters identified with the perception algorithm based upon the set of situational probabilities; and
repeat the computing of the vehicle-level failure rate and the modifying of the one or more parameters identified with the perception algorithm until the vehicle-level failure rate of the vehicle meets the predetermined safety goal.
15 . The computing device of claim 13 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to:
repeat the computing of the vehicle-level failure rate and the modifying of the one or more parameters identified with the driving policy until the vehicle-level failure rate of the vehicle meets the predetermined safety goal,
wherein the one or more parameters identified with the driving policy that are modified include at least one of (i) a minimum longitudinal distance between the vehicle and other vehicles, and (ii) a minimum lateral distance between the vehicle and other vehicles while the vehicle is driving.
16 . The computing device of claim 13 , wherein the vehicle-level failure rate represents an inverse of a mean time between failure (MTBF) of the vehicle due to the subset of perception errors resulting in a collision.
17 . The computing device of claim 13 , wherein the predefined severity threshold comprises a S2 or S3 severity according to the International Organization for Standardization (ISO) 26262 Standard.
18 . The computing device of claim 13 , wherein each respective set of predetermined driving conditions represents a combination of (i) a predetermined speed range, and (ii) a predetermined set of conditions that define a potentially dangerous traffic situation.
19 . The computing device of claim 18 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to:
compute the vehicle-level failure rate based upon a combination of the set of situational probabilities that are computed for each respective set of predetermined driving conditions.
20 . The computing device of claim 13 , wherein the second predetermined set of constraints yield only safety-relevant perception errors that may result in a collision of a predetermined level of severity that is based upon a delta-velocity at an expected collision time exceeding a predetermined velocity.
21 . The computing device of claim 13 , wherein the computer-readable instructions stored on the memory, when executed by the one or more processors, further cause the computing device to identify the perception errors by accessing sensor data from a database, and
when the sensor data stored in the sensor database is less than a threshold size, to identify the perception errors using data augmentation.
22 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by processing circuitry of a computing device, cause the computing device to:
identify perception errors as a result of execution of a perception algorithm implemented via a vehicle comprising an autonomous or a semi-autonomous vehicle, which operates in accordance with a driving policy;
provide a subset of the perception errors that (i) satisfy a first predetermined set of constraints that define the perception errors as safety-relevant to the vehicle due to the perception errors resulting in a change in a safety decision of a planning system of the vehicle, and (ii) further satisfy a second predetermined set of constraints that define the perception errors as being severe due to the perception errors resulting in a collision that exceeds a predefined severity threshold;
correlate each perception error from among the subset of perception errors to a respective set of predetermined driving conditions to determine a perception error rate per each respective set of predetermined driving conditions;
access data from a driving dataset to compute a set of situational probabilities, each respective one of the situational probabilities corresponding to a probability of the vehicle driving in conditions that match each of the respective set of predetermined driving conditions;
compute a vehicle-level failure rate of the vehicle due to the perception errors based upon the set of situational probabilities and the perception error rate per each respective set of predetermined driving conditions;
determine whether the vehicle is compliant with a predetermined safety goal based upon the vehicle-level failure rate; and
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify, via one or more processors, one or more parameters of the driving policy based upon the set of situational probabilities until the vehicle-level failure rate of the vehicle meets the predetermined safety goal.
23 . The non-transitory computer-readable medium of claim 22 , wherein the computer-readable instructions, when executed by the processing circuitry, further cause the computing device to:
when the vehicle-level failure rate of the vehicle fails to meet the predetermined safety goal, modify, via one or more processors, one or more parameters identified with the perception algorithm based upon the set of situational probabilities; and
repeat the computer-implemented method to compute the vehicle-level failure rate and to modify the one or more parameters identified with the perception algorithm until the vehicle-level failure rate of the vehicle meets the predetermined safety goal.
24 . The non-transitory computer-readable medium of claim 22 , wherein the computer-readable instructions, when executed by the processing circuitry, further cause the computing device to modify the one or more parameters of the driving policy by repeating the computer-implemented method to compute the vehicle-level failure rate and to modify the one or more parameters identified with the driving policy until the vehicle-level failure rate of the vehicle meets the predetermined safety goal, and
wherein the one or more parameters identified with the driving policy that are modified include at least one of (i) a minimum longitudinal distance between the vehicle and other vehicles, and (ii) a minimum lateral distance between the vehicle and other vehicles while the vehicle is driving.
25 . The computing device of claim 1 , wherein the predetermined safety goal comprises a threshold Mean Time Between Failure (MTBF) defined in accordance with the International Organization for Standardization (ISO) 26262 Standard.