Method and system for integrated global and distributed learning in autonomous driving vehicles

1. A method, comprising:

receiving sensor data acquired continuously by a plurality of sensors deployed on an autonomous driving vehicle, each sensor from the plurality of sensors having a sensor type from a plurality of sensor types, the sensor data providing information about a region surrounding the autonomous driving vehicle;

tracking, based on at least one object detection model residing in the autonomous driving vehicle, (1) one or more items in the region surrounding the autonomous driving vehicle, and (2) one or more features associated with at least one item from the one or more items, based on a first subset of the sensor data, the first subset of the sensor data being acquired by a first subset of at least one sensor from the plurality of sensors, the first subset of at least one sensor being of a first sensor type from the plurality of sensor types, the first subset of the sensor data being in a first modality;

labeling, automatically on-the-fly, the one or more items and the one or more features via cross modality validation of the one or more items and the one or more features based on a second subset of the sensor data in a second modality, the second subset of the sensor data being acquired by a second subset of at least one sensor from the plurality of sensors, the second subset of at least one sensor being of a second sensor type from the plurality of sensor types;

selecting, from the one or more items and after the labeling, a plurality of events of interest based on a mode of local model adaptation, the mode of local model adaptation being at least one of discrepancy based adaptation or reinforcement based adaptation; and

locally adapting, on-the-fly, the at least one object detection model based on at least one event of interest from the plurality of events of interest, the locally adapting including:

selecting, from the labeled one or more items, a first group of labeled items, each labeled item from the first group of labeled items indicating a discrepancy between a label of the labeled one or more items and a state of the one or more items in the region surrounding the autonomous driving vehicle;

selecting, from the labeled one or more items, a second group of labeled items, each labeled item from the second group of labeled items indicating a consistency between a label of the labeled one or more items and a state of the labeled one or more items in the region surrounding the autonomous driving vehicle;

determining an adaptation mode based on a model adaptation configuration; and

modifying the at least one object detection model based on the adaptation mode and at least one of:

the first group of labeled items,

the second group of labeled items, or

the first group of labeled items and the second group of labeled items; and

further comprising labeling the one or more items via cross temporal validation which comprises, for each of the one or more items:

determining an estimated label for that item;

retrieving previously labeled items associated with that item, each of the previously labeled items having an associated previous label from a plurality of previous labels;

assessing consistency between the plurality of previous labels and the estimated label;

if the estimated label is consistent with the plurality of previous labels, assigning the estimated label to that item; and

if the estimated label is inconsistent with the plurality of previous labels:

assigning the estimated label to that item and the previously labeled items if the estimated label is determined to be able to resolve the inconsistency,

assigning the previous labels to that item if the previous labels are able to resolve the inconsistency, and

assigning the estimated label to that item when the inconsistency is not resolved.

2. The method of claim 1 , wherein each of the one or more items is labeled as one of:

an object appearing in the region surrounding the autonomous driving vehicle;

a non-object; or

a non-conclusive item,

the label for each item from the one or more items having an associated confidence metric and an associated time.

3. The method of claim 1 , wherein the labeling via cross modality validation includes:

obtaining the second subset of the sensor data; and

for each of the one or more items and at each point of time from a plurality of points of time:

generating validation base data based on the second subset of the sensor data obtained at that point of time,

identifying a portion of the validation base data associated with that item,

cross validating, on-the-fly, that item based on the portion of the validation base data to generate an associated cross modality validation result,

and

labeling that item with a label based on the cross modality validation result and a time stamp consistent with that point of time.

4. The method of claim 1 , wherein, for each of the one or more items, the estimated label of that item is determined based on a result of cross modality validation with respect to that item.

5. The method of claim 1 , further comprising

sending, via a communication platform, a representation of at least one event of interest from the plurality of events of interest to a model update center to cause the model update center to derive model update information based on the at least one event of interest from the plurality of events of interest;

receiving, from the model update center, the model update information; and

updating the at least one object detection model based on the model update information.

6. A non-transitory, processor-readable medium storing instructions to cause a processor to:

receive sensor data acquired continuously by a plurality of sensors deployed on an autonomous driving vehicle, each sensor from the plurality of sensors having a sensor type from a plurality of sensor types, the sensor data providing information about a region surrounding the autonomous driving vehicle;

track based on at least one object detection model residing in the autonomous driving vehicle, (1) one or more items in the region surrounding the autonomous driving vehicle, and (2) one or more features associated with at least one item from the one or more items, based on a first subset of the sensor data, the first subset of the sensor data being acquired by a first subset of at least one sensor from the plurality of sensors, the first subset of at least one sensor being of a first sensor type from the plurality of sensor types, the first subset of the sensor data being in a first modality;

label, automatically on-the-fly, the one or more items and the one or more features via cross modality validation of the one or more items and the one or more features based on a second subset of the sensor data in a second modality, the second subset of the sensor data being acquired by a second subset of at least one sensor from the plurality of sensors, the second subset of at least one sensor being of a second sensor type from the plurality of sensor types;

select, from the one or more items and after the labeling, a plurality of events of interest based on a mode of local model adaptation, the mode of local model adaptation being at least one of discrepancy based adaptation or reinforcement based adaptation;

locally adapt, on-the-fly, the at least one object detection model based on at least one event of interest from the plurality of events of interest, by;

selecting, from the labeled one or more items, a first group of labeled items, each labeled item from the first group of labeled items indicating a discrepancy between a label of the labeled one or more items and a state of the one or more items in the region surrounding the autonomous driving vehicle;

selecting, from the labeled one or more items, a second group of labeled items, each labeled item from the second group of labeled items indicating a consistency between a label of the labeled one or more items and a state of the labeled one or more items in the region surrounding the autonomous driving vehicle;

determining an adaptation mode based on a model adaptation configuration;

modifying the at least one object detection model based on the adaptation mode and at least one of:

the first group of labeled items,

the second group of labeled items, or

the first group of labeled items and the second group of labeled items; and

wherein the instructions further include instructions to label the one or more items via cross temporal validation which comprises, for each of the one or more items:

determining an estimated label for that item;

retrieving previously labeled items associated with that item, each of the previously labeled items having an associated previous label from a plurality of previous labels;

assessing consistency between the plurality of previous labels and the estimated label;

if the estimated label is consistent with the plurality of previous labels, assigning the estimated label to that item; and

if the estimated label is inconsistent with the plurality of previous labels:

assigning the estimated label to that item and the previously labeled items if the estimated label is determined to be able to resolve the inconsistency,

assigning the previous labels to that item if the previous labels are able to resolve the inconsistency, and

assigning the estimated label to that item when the inconsistency is not resolved.

7. The non-transitory, processor-readable medium of claim 6 , wherein each of the one or more items is labeled as one of:

an object appearing in the region surrounding the autonomous driving vehicle;

a non-object; or

a non-conclusive item,

the label for each item from the one or more items having an associated confidence metric and an associated time.

8. The non-transitory, processor-readable medium of claim 6 , wherein the step of labeling via cross modality validation includes:

obtaining the second subset of the sensor data; and

for each of the one or more items and at each point of time from a plurality of points of time:

generating validation base data based on the second subset of the sensor data obtained at that point of time,

identifying a portion of the validation base data associated with that item,

cross validating, on-the-fly, that item based on the portion of the validation base data to generate an associated cross modality validation result, and

labeling that item with a label based on the cross modality validation result and a time stamp consistent with that point of time.

9. The non-transitory, processor-readable medium of claim 6 wherein, for each of the one or more items, the determining the estimated label of that item-is based on a result of cross modality validation with respect to that item.

10. The non-transitory, processor-readable medium of claim 6 , wherein the instructions further include instructions to cause the processor to:

send, via a communication platform, a representation of at least one event of interest from the plurality of events of interest to a model update center to facilitate the model update center to derive model update information based on the at least one event of interest from the plurality of events of interest;

receive, from the model update center, the model update information; and

update the at least one object detection model based on the model update information.

11. A system, comprising:

one or more sensor data receivers configured to receive sensor data acquired continuously by a plurality of sensors deployed on an autonomous driving vehicle, each sensor from the plurality of sensors having a sensor type from a plurality of sensor types, the sensor data providing information about a region surrounding the autonomous driving vehicle;

an object detection and tracking unit configured to track, based on at least one object detection model residing in the autonomous driving vehicle, ( 1 ) one or more items in the region surrounding the autonomous driving vehicle, and ( 2 ) one or more features associated with at least one item from the one or more items, based on a first subset of the sensor data, the first subset of the sensor data being acquired by a first subset of at least one sensor from the plurality of sensors, the first subset of at least one sensor being of a first sensor type from the plurality of sensor types, the first subset of the sensor data being in a first modality;

an on-the-fly data labeling unit configured to label, automatically on-the-fly, the one or more items and the one or more features via cross modality validation of the one or more items and the one or more features based on a second subset of the sensor data in a second modality,

the second subset of the sensor data being acquired by a second subset of at least one sensor from the plurality of sensors, the second subset of at least one sensor being of a second sensor type from the plurality of sensor types;

an event of interest selector configured to select, from the one or more items and after the labeling, a plurality of events of interest based on a mode of local model adaptation, the mode of local model adaptation being at least one of discrepancy based adaptation or reinforcement based adaptation;

a local model adaptation unit configured to locally adapt, on-the-fly, the at least one object detection model based on at least one event of interest from the plurality of events of interest, the local model adaptation unit including:

a local model adaptation data selector configured to:

select, from the labeled one or more items, a first group of labeled items, each labeled item from the first group of labeled items indicating a discrepancy between a label of the labeled one or more items and a state of the one or more items in the region surrounding the autonomous driving vehicle, and

select, from the labeled one or more items, a second group of labeled items, each labeled item from the second group of labeled items indicating a consistency between a label of the labeled one or more items and a state of the labeled one or more items in the region surrounding the autonomous driving vehicle;

a local model adaptation controller configured to determine an adaptation mode based on a model adaptation configuration;

at least one of:

a discrepancy based model adaptation unit configured to modify the at least one object detection model based on the first group of labeled items; or

a reinforcement based model adaptation unit configured to modify the at least one object detection model based on the second group of labeled items; and

wherein the on-the-fly data labeling unit further comprises a cross temporal validation unit configured to perform, for each of the one or more items:

obtaining an estimated label for that item, the estimated label identified based on a level of confidence in tracking that item;

retrieving previously labeled items associated with the item, each of the previously labeled items having an associated previous label from a plurality of previous labels;

assessing consistency between the plurality of previous labels and the estimated label;

if the estimated label is consistent with the plurality of previous labels, assigning the estimated label to that item; and

if the estimated label is inconsistent with the plurality of previous labels:

assigning the estimated label to that item and the previously labeled items if the estimated label is determined to be able to resolve the inconsistency,

assigning the previous labels to that item if the previous labels are able to resolve the inconsistency, and

assigning the estimated label to that item when the inconsistency is not resolved.

12. The system of claim 11 , wherein each of the one or more items is labeled as one of:

an object appearing in the region surrounding the autonomous driving vehicle;

a non-object; or

a non-conclusive item,

the label for each item from the one or more items having an associated confidence metric and an associated time.

13. The system of claim 11 , wherein the on-the-fly data labeling unit comprises:

at least one sensor data collector configured to obtain the second subset of the sensor data;

a validation base data generator configured to generate validation base data based on the second type of sensor data; and

a cross modality validation unit configured to perform, for each of the one or more items at each point of time from a plurality of points of time:

identifying a portion of the validation base data associated with that item,

cross validating, on-the-fly, that item based on the portion of the validation base data to generate an associated cross modality validation result,

and

labeling that item with a label based on the cross modality validation result and a time stamp consistent with that point of time.

14. The system of claim 11 wherein, for each of the one or more items, the estimated label of that item is determined based on a result of cross modality validation with respect to that item.

15. The system of claim 11 , further comprising:

a global model update unit configured to:

send, via a communication platform, a representation of at least one of the events of interest to a model update center to cause the model update center to derive model update information based on the at least one of the events of interest,

receive, from the model update center, the model update information, and

update the at least one object detection model based on the model update information.