METHOD AND SYSTEM FOR ON-THE-FLY OBJECT LABELING VIA CROSS TEMPORAL VALIDATION IN AUTONOMOUS DRIVING VEHICLES
The present teaching relates to method, system, medium, and implementation of in-situ perception in an autonomous driving vehicle. A plurality of types of sensor data are acquired continuously via a plurality of types of sensors deployed on the vehicle, where the plurality of types of sensor data provide information about surrounding of the vehicle. One or more items surrounding the vehicle are tracked, based on a model, from a first of the plurality of types of sensor data from a first type of the plurality of types of sensors. For a tracked item, cross temporal validation is performed by obtaining an estimated label for the item, retrieving previous labels corresponding to the item with corresponding previous time stamps, and assigning the estimated label to the item if the estimated label is validated based on previous labels to generate a labeled item. The labeled items are to be used to generate model updated information, which is then used to update the model.
1 . A method implemented on a computer having at least one processor, a storage, and a communication platform for in-situ perception in an autonomous driving vehicle, comprising:
receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;
tracking, in accordance with at least one model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle; and
for each of at least some for the one or more items, performing cross temporal validation by
obtaining an estimated label for the item,
retrieving previous labels corresponding to the item, wherein each of the previous labels has a corresponding previous time stamp, and
assigning the estimated label to the item if the estimated label is validated based on previous labels to generate a labeled item, wherein
the labeled at least some items are to be used to generate model updated information, which is then used to update the at least one model.
2 . The method of claim 1 , wherein each of the at least some of the one or more items is labeled as one of:
an object appearing in the surrounding of the vehicle;
a non-object; and
a non-conclusive item, wherein
each label is provided in conjunction with a measure indicative of a level of confidence in the label, and
each label associated with an item is provided with respect to a time.
3 . The method of claim 1 , wherein the estimated label is generated based on cross modality validation.
4 . The method of claim 3 , wherein the cross modality validation is achieved by:
obtaining a second of the plurality of types of sensor data from at least one of a second type of the plurality of types of sensors;
generating validation base data based on the obtained second type of sensor data; and
estimating, for each of the at least some of the one or more items at each point of time, the estimated label by
registering the item tracked at the point of time with a portion of the validation base data acquired at the point of time,
cross validating, on-the-fly, the item at the point of time based on the portion of the validation base data to generate a corresponding cross modality validation result, and
generating the estimated label with a time stamp consistent with the point of time for the item based on the cross modality validation result.
5 . The method of claim 1 , further comprising:
for each of the at least some of the one or more items, when the estimated label is inconsistent with the previous labels,
assigning, if the estimated label can be used to resolve the inconsistency, the estimated label to the previously labeled items and the item to generate the labeled item,
assigning, if the previous labels are consistent and can be used to resolve the inconsistency, the previous labels to the item to generate the labeled item, and
assigning, when the inconsistency is not resolved, the estimated label to the item to generate the labeled item.
6 . The method of claim 1 , further comprising
receiving, from a model update center, the model update information derived based on the labeled at least some items; and
updating the at least one model based on the model update information.
7 . Machine readable and non-transitory medium having data recorded thereon for in-situ perception in an autonomous driving vehicle, wherein the data, once read by the machine, cause the machine to perform the following:
receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;
tracking, in accordance with at least one model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle; and
for each of at least some for the one or more items, performing cross temporal validation by
obtaining an estimated label for the item,
retrieving previous labels corresponding to the item, wherein each of the previous labels has a corresponding previous time stamp, and
assigning the estimated label to the item if the estimated label is validated based on previous labels to generate a labeled item, wherein
the labeled at least some items are to be used to generate model updated information, which is then used to update the at least one model.
8 . The medium of claim 7 , wherein each of the at least some of the one or more items is labeled as one of:
an object appearing in the surrounding of the vehicle;
a non-object; and
a non-conclusive item, wherein
each label is provided in conjunction with a measure indicative of a level of confidence in the label, and
each label associated with an item is provided with respect to a time.
9 . The medium of claim 7 , wherein the estimated label is generated based on cross modality validation.
10 . The medium of claim 9 , wherein the cross modality validation is achieved by:
obtaining a second of the plurality of types of sensor data from at least one of a second type of the plurality of types of sensors;
generating validation base data based on the obtained second type of sensor data; and
estimating, for each of the at least some of the one or more items at each point of time, the estimated label by
registering the item tracked at the point of time with a portion of the validation base data acquired at the point of time,
cross validating, on-the-fly, the item at the point of time based on the portion of the validation base data to generate a corresponding cross modality validation result, and
generating the estimated label with a time stamp consistent with the point of time for the item based on the cross modality validation result.
11 . The medium of claim 7 , wherein, the data, when read by the machine, cause the machine to further perform:
for each of the at least some of the one or more items, when the estimated label is inconsistent with the previous labels,
assigning, if the estimated label can be used to resolve the inconsistency, the estimated label to the previously labeled items and the item to generate the labeled item,
assigning, if the previous labels are consistent and can be used to resolve the inconsistency, the previous labels to the item to generate the labeled item, and
assigning, when the inconsistency is not resolved, the estimated label to the item to generate the labeled item.
12 . The medium of claim 7 , wherein, the data, when read by the machine, cause the machine to further perform:
receiving, from a model update center, the model update information derived based on the labeled at least some items; and
updating the at least one model based on the model update information.
13 . A system for in-situ perception in an autonomous driving vehicle, comprising:
one or more sensor data receivers configured for receiving a plurality of types of sensor data acquired continuously by a plurality of types of sensors deployed on the vehicle, wherein the plurality of types of sensor data provide information about surrounding of the vehicle;
an object detection & tracking unit configured for tracking, in accordance with at least one model, one or more items from a first of the plurality of types of sensor data acquired by one or more of a first type of the plurality of types of sensors, wherein the one or more items appear in the surrounding of the vehicle; and
a cross temporal validation unit configured for, for each of at least some for the one or more items,
obtaining an estimated label for the item,
retrieving previous labels corresponding to the item, wherein each of the previous labels has a corresponding previous time stamp, and
assigning the estimated label to the item if the estimated label is validated based on previous labels to generate a labeled item, wherein
the labeled at least some items is to be used to generate model updated information, which is then used to update the at least one model.
14 . The system of claim 13 , wherein each of the at least some of the one or more items is labeled as one of:
an object appearing in the surrounding of the vehicle;
a non-object; and
a non-conclusive item, wherein
each label is provided in conjunction with a measure indicative of a level of confidence in the label, and
each label associated with an item is provided with respect to a time.
15 . The system of claim 13 , further comprising a cross modality validation unit configured for generating the estimated label.
16 . The system of claim 15 , wherein the cross modality validation unit is configured for:
obtaining a second of the plurality of types of sensor data from at least one of a second type of the plurality of types of sensors;
generating validation base data based on the obtained second type of sensor data; and
estimating, for each of the at least some of the one or more items at each point of time, the estimated label by
registering the item tracked at the point of time with a portion of the validation base data acquired at the point of time,
cross validating, on-the-fly, the item at the point of time based on the portion of the validation base data to generate a corresponding cross modality validation result, and
generating the estimated label with a time stamp consistent with the point of time for the item based on the cross modality validation result.
17 . The system of claim 13 , wherein the cross temporal validation unit is further configured for,
for each of the at least some of the one or more items, when the estimated label is inconsistent with the previous labels,
assigning, if the estimated label can be used to resolve the inconsistency, the estimated label to the previously labeled items and the item to generate the labeled item,
assigning, if the previous labels are consistent and can be used to resolve the inconsistency, the previous labels to the item to generate the labeled item, and
assigning, when the inconsistency is not resolved, the estimated label to the item to generate the labeled item.
18 . The system of claim 13 , further comprising a global model update unit configured for:
receiving, from a model update center, the model update information derived based on the labeled at least some items; and
updating the at least one model based on the model update information.