Retraining neural network model based on sensor data filtered for corner case
A computer-implemented method comprises: receiving sensor data collected during operation of a vehicle, the sensor data including images from at least one camera on the vehicle and other sensor data from at least one other sensor on the vehicle; identifying, based on the sensor data, a corner case for a first neural network model of an advanced driver assistance system (ADAS) of the vehicle; filtering the images to identify a subset of the images depicting the corner case, the filtering performed using a second neural network model; labeling the subset of the images with regard to the corner case, wherein a remainder of the images are not labeled; and retraining the first neural network model using the labeled subset of the images and a subset of the other sensor data corresponding to the subset of the images.
1 . A computer-implemented method comprising:
receiving sensor data collected during operation of a vehicle, the sensor data including images from at least one camera on the vehicle and other sensor data from at least one other sensor on the vehicle;
identifying, based on the sensor data, a corner case for a first neural network model of an advanced driver assistance system (ADAS) of the vehicle, the corner case being identified based on being a feature or phenomenon that the first neural network model will encounter only less often than about 5% of the time in training data provided to the first neural network model;
filtering the images to identify a subset of the images depicting the corner case, the filtering performed using a second neural network model;
labeling the subset of the images with regard to the corner case, wherein a remainder of the images are not labeled; and
retraining the first neural network model using the labeled subset of the images and a subset of the other sensor data corresponding to the subset of the images.
2 . The computer-implemented method of claim 1 , wherein identifying the corner case comprises detecting a discrepancy between first and second neural network outputs, the first output generated based on a first sensor of the vehicle, and the second output generated based on a second sensor of the vehicle.
3 . The computer-implemented method of claim 2 , wherein the first output is a camera-based three-dimensional (3D) prediction, and wherein the second output is a lidar-based 3D prediction.
4 . The computer-implemented method of claim 3 , wherein retraining the first neural network model comprises retraining a model for the camera-based 3D prediction.
5 . The computer-implemented method of claim 3 , wherein retraining the first neural network model comprises retraining a model for the lidar-based 3D prediction.
6 . The computer-implemented method of claim 1 , wherein the corner case for the first neural network model comprises an event.
7 . The computer-implemented method of claim 1 , wherein the corner case for the first neural network model comprises a specification of an object.
8 . The computer-implemented method of claim 7 , wherein the specification identifies a type of vehicle.
9 . The computer-implemented method of claim 1 , wherein the first neural network model generates a confidence score while the ADAS is controlling the vehicle, and wherein identifying the corner case comprises determining that the confidence score is below a threshold value.
10 . The computer-implemented method of claim 1 , wherein identifying the corner case comprises detecting a driver takeover from the ADAS.
11 . The computer-implemented method of claim 1 , wherein identifying the corner case comprises detecting that a driving maneuver by the ADAS exceeds a threshold value.
12 . The computer-implemented method of claim 1 , wherein the filtering is only performed on the images of the sensor data.
13 . The computer-implemented method of claim 1 , wherein the first and second neural network models are one and the same.
14 . The computer-implemented method of claim 1 , wherein the first and second neural network models are separate from each other.
15 . The computer-implemented method of claim 1 , wherein at least the filtering is performed onboard the vehicle.
16 . The computer-implemented method of claim 15 , further comprising storing triggers on the vehicle, wherein filtering is based on the triggers.
17 . The computer-implemented method of claim 16 , further comprising providing a new trigger to the vehicle by way of an over the air update.
18 . The computer-implemented method of claim 1 , further comprising providing the retrained first neural network model to the vehicle by way of an over the air update.
19 . The computer-implemented method of claim 1 , wherein at least the filtering and retraining are performed offboard the vehicle.
20 . A computer-readable medium including instructions that when executed by a processor cause the processor configured to perform the method of claim 1 .