Systems and methods for validating a localization system of an autonomous vehicle

1 . A computer-implemented method for validating a localization system of an autonomous vehicle, comprising:

(a) obtaining log data descriptive of a plurality of environmental and operational conditions associated with operation of the autonomous vehicle in an environment;

(b) augmenting the log data with one or more simulated anomalies;

(c) simulating operation of the localization system using the augmented data as input;

(d) obtaining simulation result data descriptive of a state of the localization system subsequent to (c); and

(e) determining a metric associated with the localization system based on the simulation result data;

wherein the metric is determined based on one or more statistical correlations learned through simulation of localization scenarios.

2 . The computer-implemented method of claim 1 , wherein the log data comprises localization result data descriptive of an output or state of the localization system during operation of the autonomous vehicle in the environment.

3 . The computer-implemented method of claim 2 , wherein determining the metric associated with the localization system is based on a comparison between the localization result data and simulation result data.

4 . The computer-implemented method of claim 1 , wherein (c) comprises dropping one or more inputs to the localization system.

5 . The computer-implemented method of claim 4 , wherein the one or more inputs comprise outputs of at least one of: (i) a lane alignment system, (ii) a surfel registration system, (iii) a sensor velocity system, (iv) an IMU, (v) one or more wheel encoders, (vi) a GNSS, (vii) a RADAR system, or (viii) a LIDAR system.

6 . The computer-implemented of claim 1 , wherein (c) comprises disabling at least one of: (i) a lane alignment system, (ii) a surfel registration system, (iii) a sensor velocity system, (iv) an IMU, (v) one or more wheel encoders, (vi) a GNSS, (vii) a RADAR system, or (viii) a LIDAR system.

7 . The computer-implemented method of claim 1 , wherein (c) comprises degrading one or more inputs to the localization system.

8 . The computer-implemented method of claim 7 , wherein degrading one or more inputs to the localization system comprises at least one of: (i) disabling outputs of one or more sensors in a sensor system; (ii) utilizing one or more incorrect calibrations on a sensor; (iii) simulating one or more occlusions; (iv) simulating latency at the one or more inputs; (v) simulating one or more boundary conditions; or (vi) injecting a checkpoint fault.

9 . The computer-implemented method of claim 1 , wherein (c) comprises augmenting map data provided to the localization system.

10 . The computer-implemented method of claim 9 , wherein the augmented map data comprises one or more of out-of-date map data, map data representing new construction or vegetation, misaligned map data, map data comprising missing surfel registrations, or map data depicting inaccurate travelway regions.

11 . The computer-implemented method of claim 1 , wherein the simulation result data is descriptive of one or more anomalous outputs of the localization filter.

12 . The computer-implemented method of claim 1 , wherein (e) comprises:

determining one or more subsystem metrics associated with one or more subsystems of the localization system; and

determining the metric associated with the localization filter based on the one or more subsystem metrics.

13 . The computer-implemented method of claim 1 , wherein the metric is indicative of whether the localization system localized the autonomous vehicle with a threshold confidence.

14 . The computer-implemented method of claim 1 , wherein the method further comprises:

(f) determining that the metric associated with the localization filter satisfies a threshold; and

(g) in response to determining that the metric associated with the localization filter satisfies the threshold, validating the localization filter.

15 . An autonomous vehicle control system, comprising:

one or more processors; and

one or more non-transitory, computer-readable media storing instructions that cause the one or more processors to perform operations comprising:

(a) obtaining log data descriptive of a plurality of environmental and operational conditions associated with operation of the autonomous vehicle in an environment;

(b) augmenting the log data with one or more simulated anomalies;

(c) simulating operation of the localization system using the augmented data as input;

(d) obtaining simulation result data descriptive of a state of the localization system subsequent to (c); and

(e) determining a metric associated with the localization system based on the simulation result data;

wherein the metric is determined based on one or more statistical correlations learned through simulation of localization scenarios.

16 . The autonomous vehicle control system of claim 15 , wherein the log data comprises localization result data descriptive of an output or state of the localization system during operation of the autonomous vehicle in the environment.

17 . The autonomous vehicle control system of claim 16 , wherein determining the metric associated with the localization system is based on a comparison between a result of one or more first testing instances and the simulation result data.

18 . The autonomous vehicle control system of claim 15 , wherein (c) comprises dropping one or more inputs to the localization system, wherein the one or more inputs comprise outputs of at least one of: (i) a lane alignment system, (ii) a surfel registration system, (iii) a sensor velocity system, (iv) an IMU, (v) one or more wheel encoders, (vi) a GNSS, (vii) a RADAR system, or (viii) a LIDAR system.

19 . An autonomous vehicle, comprising:

one or more processors; and

one or more non-transitory, computer-readable media storing instructions that cause the one or more processors to perform operations comprising:

(a) obtaining log data descriptive of a plurality of environmental and operational conditions associated with operation of the autonomous vehicle in an environment;

(b) augmenting the log data with one or more simulated anomalies;

(c) simulating operation of the localization system using the augmented data as input;

(d) obtaining simulation result data descriptive of a state of the localization system subsequent to (c); and

(e) determining a metric associated with the localization system based on the simulation result data;

wherein the metric is determined based on one or more statistical correlations learned through simulation of localization scenarios.

20 . The autonomous vehicle of claim 19 , wherein the log data comprises localization result data descriptive of an output or state of the localization system during operation of the autonomous vehicle in the environment.