Method, apparatus, and system for in-vehicle data selection for feature detection model creation and maintenance
An approach is provided for selecting training observations for machine learning models. The approach involves determining a first distribution of a plurality of features observed in the training data set, and a second distribution of the plurality of features observed in the candidate pool of observations. The approach further involves selecting one or more observations in the candidate pool of observations for annotation based on the first distribution and the second distribution. The approach further involves adding the one or more observations to the training data set after annotation. The training data set is used for training the machine learning model.
1. A computer-implemented method for use in an in-vehicle feature detection device comprising an initial machine learning model trained to perform feature detection, the method comprising:
processing, by the initial machine learning model, sensor data collected by a vehicle to output a detected feature and a confidence metric for the detected feature, wherein the confidence metric represents uncertainty associated with road features or objects detected from sensor readings;
selecting a portion of the sensor data having the confidence metric below a confidence threshold;
transmitting the portion of the sensor data to an external server over a communication network;
receiving a second machine learning model over the communication network from the external server, wherein the second machine learning model is created or re-trained to predict the detected feature associated with the confidence metric below the confidence threshold at a desired or configured level of accuracy; and
replacing the initial machine learning model with the second machine learning model.
2. The method of claim 1 , further comprising:
prior to transmitting the portion of the sensor data, marking the sensor data with a data record indicating that the sensor data is novel based on determining that the confidence metric is below the confidence threshold.
3. The method of claim 2 , wherein the external server determines whether to store or process the sensor data based on the marking of the sensor data.
4. The method of claim 1 , further comprising:
determining that a privacy policy associated with the vehicle's location, the in-vehicle feature detection device, or a combination thereof restricts the transmitting of the sensor data from the vehicle; and
transmitting other information associated with the sensor data from the vehicle to the mapping platform in place of the sensor data based on determining that the confidence metric is below the confidence threshold.
5. The method of claim 4 , wherein the other information includes contextual information associated with a collection of the sensor data, the vehicle, an environment surrounding the vehicle, or a combination thereof.
6. The method of claim 1 , wherein the transmitting of the portion of the sensor data by the in-vehicle feature detection device is further based on determining that a location where the sensor data is collected corresponds to a requested location specified by a mapping platform, wherein the requested location is a geo-fenced region of interest.
7. The method of claim 1 , wherein the confidence threshold is based, at least in part, on a data storage threshold, a transmission bandwidth threshold, a data sparsity at a target location, or a combination thereof.
8. The method of claim 1 , wherein a raw sensor data is not transmitted from the vehicle based on determining that the confidence metric is above the confidence threshold.
9. An apparatus comprising:
at least one processor; and
at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
receive sensor data from an in-vehicle feature detection device over a communication network, wherein the sensor data comprises a detected feature that has a confidence metric below a confidence threshold, wherein the confidence metric represents uncertainty associated with road features or objects detected from sensor readings;
process the sensor data and identify the detected feature;
create or re-train a feature detection model, wherein the feature detection model generates a prediction of the detected feature at a desired or configured level of accuracy; and
transmit the feature detection model to the in-vehicle feature detection device over the communication network to replace an initial feature detection model used by the in-vehicle feature detection device.
10. The apparatus of claim 9 , wherein the apparatus is further caused to:
transmit a request to the vehicle to capture data from a geo-fenced region of interest, wherein the sensor data is received in response to the request; and
wherein the sensor data is collected by the vehicle from the geo-fenced region of interest.
11. The apparatus of claim 10 , wherein the geo-fenced region of interest is associated with a historical feature detection performance that is below a threshold performance.
12. The apparatus of claim 11 , wherein the apparatus is further caused to:
prior to transmitting the sensor data, mark as novel the portion of sensor data wherein the confidence metric is below the confidence threshold;
store the sensor data marked as novel in a permanent storage; and
store the sensor data not marked as novel in a temporary storage.
13. The apparatus of claim 9 , wherein the processing of the sensor data to create or re-train the feature detection model includes annotating the sensor data that includes the feature detected with the confidence metric below the confidence threshold with one or more feature labels prior to creation or re-training.
14. The apparatus of claim 9 , wherein the apparatus is further caused to:
receive other information over the communication network from the vehicle, wherein:
other information comprises one or more conditions associated with collecting the sensor data in place of the sensor data when a privacy policy of the vehicle, the in-vehicle feature detection device, or a combination thereof restricts transmitting the sensor data externally from the vehicle, and
the one or more conditions comprise road conditions, weather conditions, or a combination thereof; and
transmit a request for the vehicle or another vehicle to capture additional sensor data under the one or more conditions.
15. A non-transitory computer-readable storage medium carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to perform:
processing, by an initial machine learning model trained to perform feature detection, sensor data collected by a vehicle to output a detected feature and a confidence metric for the detected feature, wherein the confidence metric represents uncertainty associated with the detected feature;
selecting a portion of the sensor data having the confidence metric below a confidence threshold;
transmitting the portion of the sensor data to an external server over a communication network;
receiving a second machine learning model over the communication network from the external server, wherein the second machine learning model is created or re-trained to predict the detected feature associated with the confidence metric below the confidence threshold at a desired or configured level of accuracy; and
replacing the initial machine learning model with the second machine learning model.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the apparatus is further caused to perform:
determining that a privacy policy associated with the vehicle's location, the in-vehicle feature detection device, or a combination thereof restricts the transmitting of the portion of the sensor data from the vehicle; and
transmitting other information associated with the portion of the sensor data from the vehicle to the external server in place of the sensor data based on determining that the confidence metric is below the confidence threshold.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the transmitting of the portion of the sensor data, by the in-vehicle feature detection device, is further based on determining that a location where the sensor data is collected corresponds to a requested location specified by a mapping platform, wherein the requested location is a geo-fenced region of interest.
18. The method of claim 1 , wherein the second machine learning model is created or re-trained by a mapping platform using a learner module that:
feeds feature sets from a labeled sensor data set into the feature detection model to compute a predicted matching probability and a predicted matching feature using an initial set of model parameters;
compares a predicted matching probability and the predicted matching feature to ground truth data in the labeled sensor data set; and
adjusts the model parameters until the machine learning model generates the predicted matching feature at a desired or configured level of accuracy.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the second feature detection model is created or re-trained by a mapping platform using a learner module that:
feeds feature sets from a labeled sensor data set into the feature detection model to compute a predicted matching probability and a predicted matching feature using an initial set of model parameters;
compares a predicted matching probability and the predicted matching feature to ground truth data in the labeled sensor data set; and
adjusts the model parameters until the machine learning model generates the predicted matching feature at a desired or configured level of accuracy.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the other information includes contextual information associated with a collection of the sensor data, the vehicle, an environment surrounding the vehicle, or a combination thereof.