Method and system for distributed learning and adaptation in autonomous driving vehicles
The present teaching relates to system, method, medium for in-situ perception in an autonomous driving vehicle. A plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle are first received, where the plurality of types of sensor data provide information about surrounding of the vehicle. Based on at least one model, one or more items are tracked from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle. At least some of the one or more items are then automatically labeled on-the-fly via either cross modality validation or cross temporal validation of the one or more items and are used to locally adapt, on-the-fly, the at least one model in the vehicle.
1. A computer-implemented method comprising:
selecting, by a computing system, first candidate training data associated with a first event of interest relating to a discrepancy between labels of an item detected in a region surrounding a vehicle, wherein the discrepancy reflects inconsistency between labels and is associated with a first label for the item based on sensor data captured at a first time and a second label of the item based on sensor data captured at a second time, the first label and the second label included in the first candidate training data, wherein
the first label of the item is i) generated by a model that detected the item and ii) associated with a false negative detection of the item,
the second label of the item is i) generated by the model that detected the item, ii) not associated with a false negative detection of the item, and iii) utilized to train the model for navigation, and
determination of the false negative detection of the item is based on a first confidence score associated the first label that is lower than a second confidence score associated with the second label;
and
adapting on-the-fly, by the computing system, the model for navigation of the vehicle based on at least one of the first candidate training data, including the first label and the second label.
2. The computer-implemented method of claim 1 , wherein the discrepancy is based on labels associated with detections having different confidence scores.
3. The computer-implemented method of claim 1 , further comprising, based on the first candidate training data associated with the first event of interest relating to the discrepancy between labels,
modifying labels associated with previous detections of the item.
4. The computer-implemented method of claim 1 , further comprising:
generating at least some of the labels through cross modality validation based on sensor data in a first modality associated with a first sensor and sensor data in a second modality associated with a second sensor.
5. The computer-implemented method of claim 4 , wherein the first sensor is a passive sensor and the second sensor is an active sensor, detection of the item considered validated when the sensor data in the second modality confirms at least one feature associated with the item as estimated by sensor data in the first modality.
6. The computer-implemented method of claim 5 , wherein the at least one feature associated with the item comprises at least one of a size of the item, a depth of the item, and a texture of the item.
7. The computer-implemented method of claim 1 , wherein
the first label and the second label are generated through cross temporal validation based on sensor data captured by a first type of sensor.
8. The computer-implemented method of claim 7 , wherein a label based on sensor data captured at the second time is associated with a confidence level higher than a confidence level associated with a label based on sensor data captured at the first time, wherein the method further comprises:
self-correcting the label based on sensor data captured at the first time with the label based on sensor data captured at the second time.
9. The computer-implemented method of claim 1 , wherein the first candidate training data associated with the first event of interest relating to the discrepancy between labels and second candidate training data associated with a second event of interest relating to the consistency between labels are stored separately at the vehicle, and the adapting is performed locally at the vehicle.
10. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform operations comprising:
selecting first candidate training data associated with a first event of interest relating to a discrepancy between labels of an item detected in a region surrounding a vehicle, wherein the discrepancy reflects inconsistency between labels and is associated with a first label for the item based on sensor data captured at a first time and a second label of the item based on sensor data captured at a second time, the first label and the second label included in the first candidate training data, wherein
the first label of the item is i) generated by a model that detected the item and ii) associated with a false negative detection of the item
the second label of the item is i) generated by the model that detected the item, ii) not associated with a false negative detection of the item, and iii) utilized to train the model for navigation, and
determination of the false negative detection of the item is based on a first confidence score associated the first label that is lower than a second confidence score associated with the second label;
and
adapting on-the-fly the model for navigation of the vehicle based on at least one of the first candidate training data, including the first label and the second label.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the discrepancy is based on labels associated with detections having different confidence scores.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the operations further comprise, based on the first candidate training data associated with the first event of interest relating to the discrepancy between labels,
modifying labels associated with previous detections of the item.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the operations further comprise:
generating at least some of the labels through cross modality validation based on sensor data in a first modality associated with a first sensor and sensor data in a second modality associated with a second sensor.
14. The non-transitory computer-readable storage medium of claim 10 , wherein the operations further comprise:
generating at least some of the labels through cross temporal validation based on sensor data captured at a first time and sensor data captured at a second time.
15. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
selecting first candidate training data associated with a first event of interest relating to a discrepancy between labels of an item detected in a region surrounding a vehicle, wherein the discrepancy reflects inconsistency between labels and is associated with a first label for the item based on sensor data captured at a first time and a second label of the item based on sensor data captured at a second time, the first label and the second label included in the first candidate training data, wherein
the first label of the item is i) generated by a model that detected the item and ii) associated with a false negative detection of the item and
the second label of the item is i) generated by the model that detected the item, ii) not associated with a false negative detection of the item, and iii) utilized to train the model for navigation, and
determination of the false negative detection of the item is based on a first confidence score associated the first label that is lower than a second confidence score associated with the second label;
and
adapting on-the-fly the model for navigation of the vehicle based on at least one of the first candidate training data, including the first label and the second label.
16. The system of claim 15 , wherein the discrepancy is based on labels associated with detections having different confidence scores.
17. The system of claim 15 , wherein the operations further comprise, based on the first candidate training data associated with the first event of interest relating to the discrepancy between labels:
modifying labels associated with previous detections of the item.
18. The system of claim 15 , wherein the operations further comprise:
generating at least some of the labels through cross modality validation based on sensor data in a first modality associated with a first sensor and sensor data in a second modality associated with a second sensor.
19. The system of claim 15 , wherein the operations further comprise:
generating at least some of the labels through cross temporal validation based on sensor data captured at a first time and sensor data captured at a second time.
20. The system of claim 15 , wherein the first confidence score associated the first label and the second confidence score associated with the second label are determined by the model.