Method and system for closed loop perception in autonomous driving vehicles

1. A method implemented on a computer having at least one processor, a storage, and a communication platform for in-situ perception in an autonomous driving vehicle, comprising:

receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;

tracking, in accordance with at least one object detection model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle;

labeling, automatically on-the-fly, the one or more items via at least one of cross modality validation and cross temporal validation of the one or more items, wherein a label for each of the one or more items is associated with a confidence score;

adapting locally the at least one object detection model on-the-fly using a first set of events of interest identified from the one or more labeled items based on associated confidence scores;

sending, via the communication platform, a second set of events of interest, selected from the one or more labeled items, to a model update center, which adapts at least one global object detection model based on second sets of events of interest from a plurality of autonomous driving vehicles in a fleet;

receiving, from the model update center, model update information, generated via adapting the at least one global object detection model, for updating the at least one object detection model; and

updating the at least one object detection model based on the received model update information.

2. The method of claim 1 , wherein each of the one or more items is labeled as one of:

an object appearing in the surrounding of the vehicle;

a non-object; and

a non-conclusive item, wherein

each label is provided in conjunction with the confidence score, which is a measure indicative of a level of confidence in the label, and

each label associated with an item is provided with respect to a time.

3. The method of claim 1 , wherein the step of labeling via cross modality validation comprises:

obtaining a second of the plurality of types of sensor data from at least one of a second type of the plurality of types of sensors, and

generating validation base data based on the obtained second type of sensor data;

for each of the one or more items at each point of time,

registering the item tracked at the point of time with a portion of the validation base data acquired at the point of time,

cross validating, on-the-fly, the item at the point of time based on the portion of the validation base data to generate a corresponding cross modality validation result,

assessing the cross modality validation result with respect to the item, and

labeling the item based on a label corresponding to the cross modality validation result with a time stamp corresponding to the point of time.

4. The method of claim 1 , wherein the step of labeling via cross temporal validation comprises, for each of the one or more items,

determining a label for the item estimated based on the at least one object detection model;

retrieving previously labeled items corresponding to the item, wherein each of the previously labeled items has a corresponding previous label;

if the estimated label is the same as the previous labels, assigning the estimated label to the item.

5. The method of claim 4 , wherein the estimated label of the item is determined based on a result of cross modality validation with respect to the item.

6. A machine readable and non-transitory medium having data recorded thereon for in-situ perception in an autonomous driving vehicle, wherein the data, once read by the machine, cause the machine to perform the following:

receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;

tracking, in accordance with at least one object detection model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle;

labeling, automatically on-the-fly, the one or more items via at least one of cross modality validation and cross temporal validation of the one or more items, wherein a label for each of the one or more items is associated with a confidence score;

adapting locally the at least one object detection model on-the-fly using a first set of events of interest identified from the one or more labeled items based on associated confidence scores;

sending, via the communication platform, a second set of events of interest, selected from the one or more labeled items, to a model update center, which adapts at least one global object detection model based on second sets of events of interest from a plurality of autonomous driving vehicles in a fleet;

receiving, from the model update center, model update information, generated via adapting the at least one global object detection model, for updating the at least one object detection model; and

updating the at least one object detection model based on the received model update information.

7. The medium of claim 6 , wherein each of the one or more items is labeled as one of:

an object appearing in the surrounding of the vehicle;

a non-object; and

a non-conclusive item, wherein

each label is provided in conjunction with the confidence score, which is a measure indicative of a level of confidence in the label, and

each label associated with an item is provided with respect to a time.

8. The medium of claim 6 , wherein the step of labeling via cross modality validation comprises:

obtaining a second of the plurality of types of sensor data from at least one of a second type of the plurality of types of sensors, and

generating validation base data based on the obtained second type of sensor data;

for each of the one or more items at each point of time,

registering the item tracked at the point of time with a portion of the validation base data acquired at the point of time,

cross validating, on-the-fly, the item at the point of time based on the portion of the validation base data to generate a corresponding cross modality validation result,

assessing the cross modality validation result with respect to the item, and

labeling the item based on a label corresponding to the cross modality validation result with a time stamp corresponding to the point of time.

9. The medium of claim 6 , wherein the step of labeling via cross temporal validation comprises, for each of the one or more items,

determining a label for the item estimated based on the at least one object detection model;

retrieving previously labeled items corresponding to the item, wherein each of the previously labeled items has a corresponding previous label;

if the estimated label is the same as consistent with the previous labels, assigning the estimated label to the item.

10. The medium of claim 9 , wherein the estimated label of the item is determined based on a result of cross modality validation with respect to the item.

11. A system for in-situ perception in an autonomous driving vehicle, comprising:

one or more sensor data collectors implemented by a processor and configured for receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;

an object detection & tracking unit implemented by the processor and configured for tracking, in accordance with at least one object detection model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle;

an on-the-fly data labeling unit implemented by the processor and configured for

labeling, automatically on-the-fly, the one or more items via at least one of cross modality validation and cross temporal validation of the one or more items, wherein a label for each of the one or more items is associated with a confidence score,

adapting locally the at least one object detection model on-the-fly using a first set of events of interest identified from the one or more labeled items based on associated confidence scores;

a global model update unit implemented by the processor and configured for

sending, via the communication platform, a set of second set of events of interest, selected from the one or more labeled items, to a model update center, which adapts at least one global object detection model based on second sets of events of interest from a plurality of autonomous driving vehicles in a fleet,

receiving, from the model update center, model update information, generated via adapting the at least one global object detection model, for updating the at least one object detection model, and

updating the at least one object detection model based on the received model update information.

12. The system of claim 11 , wherein each of the of the one or more items is labeled as one of:

an object appearing in the surrounding of the vehicle;

a non-object; and

a non-conclusive item, wherein

each label is provided in conjunction with the confidence score, which is a measure indicative of a level of confidence in the label, and

each label associated with an item is provided with respect to a time.

13. The system of claim 11 , wherein the on-the-fly data labeling unit comprises:

at least one sensor data collector configured for obtaining a second of the plurality of types of sensor data from at least one of a second type of the plurality of types of sensors;

a validation base data generator configured for generating validation base data based on the obtained second type of sensor data;

a cross modality validation unit configured for performing the following for each of the one or more items at each point of time,

registering the item tracked at the point of time with a portion of the validation base data acquired at the point of time,

cross validating, on-the-fly, the item at the point of time based on the portion of the validation base data to generate a corresponding cross modality validation result,

assessing the cross modality validation result with respect to the item, and

labeling the item based on a label corresponding to the cross modality validation result with a time stamp corresponding to the point of time.

14. The system of claim 13 , wherein the on-the-fly data labeling unit further comprises a cross temporal validation unit, which is configured for performing, for each of the one or more items,

obtaining a label for the item estimated based on the at least one object detection model;

retrieving previously labeled items corresponding to the item, wherein each of the previously labeled items has a corresponding previous label;

if the estimated label is the same as consistent with the previous labels, assigning the estimated label to the item.

15. The system of claim 14 , wherein the estimated label of the item is determined based on a result of cross modality validation with respect to the item.