Determining confident data samples for machine learning models on unseen data
Techniques are provided for determining confident data samples for machine learning (ML) models on unseen data. In one embodiment, a method is provided that comprises extracting, by a system comprising a processor, a feature vector for a data sample based on projection of the data sample onto a standard feature space. The method further comprises processing, by the system, the feature vector using an outlier detection model to determine whether the data sample is within a scope of a training dataset used to train a machine learning model, wherein the outlier detection model was trained using features extracted from the training dataset based on projection of data samples included in the training dataset onto the standard feature space.
1. A method, comprising:
extracting, by a system comprising a processor, a feature vector for a data sample based on projection of the data sample onto a standard feature space;
processing, by the system, the feature vector using an outlier detection model to determine whether the data sample is within a scope of a training dataset used to train a machine learning model, wherein the outlier detection model was trained using features extracted from the training dataset based on projection of data samples included in the training dataset onto the standard feature space;
generating, by the system, a confidence score for the data sample using the outlier detection model, wherein the confidence score represents a measure of confidence in the machine learning model to generate an accurate inference on the data sample; and
classifying, by the system, the data sample as an outlier data sample or an inlier data sample based on the confidence score.
2. The method of claim 1 , wherein the outlier detection model classifies the data sample as an outlier data sample or an inlier data sample.
3. The method of claim 1 , wherein the data samples included in the training dataset that were projected onto the standard feature space were selected based on a determination that the machine learning model generated correct inferences on the data samples during training of the machine learning model.
4. The method of claim 1 , further comprising:
generating, by the system, an outlier detection notification for the data sample based on classification of the data sample as an outlier data sample.
5. The method of claim 1 , further comprising:
sending, by the system, the data sample for manual annotation and review based on classification of the data sample as an outlier data sample.
6. The method of claim 1 , further comprising:
applying, by the system, the machine learning model to the data sample to generate an inference result based on classification of the data sample as an inlier data sample.
7. The method of claim 1 , wherein the standard feature space comprises a plurality of images with annotated visual features and wherein the data sample and the data samples respectively comprise images.
8. The method of claim 1 , wherein the extracting comprises employing a feature extraction network trained on the standard feature space.
9. The method of claim 8 , wherein the standard feature space comprises an ImageNet feature space.
10. The method of claim 1 , wherein the machine learning model comprises a deep neural network model.
11. The method of claim 1 , wherein the outlier detection model employs an isolation forest (IF) outlier detection method or a one-class support vector machine (OCSVM) method.
12. A system, comprising:
a memory that stores computer executable components; and
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
a feature extraction component that extracts a feature vector for a data sample based on projection of the data sample onto a standard feature space; and
an outlier detection component that processes the feature vector using an outlier detection model to determine whether the data sample is within a scope of a training dataset used to train a machine learning model, wherein the outlier detection model was trained using features extracted from the training dataset based on projection of data samples included in the training dataset onto the standard feature space, and determines a confidence score for the data sample using the outlier detection model and classifies the data sample as an outlier data sample or an inlier data sample based on the confidence score, wherein the confidence score represents a measure of confidence in the machine learning model to generate an accurate inference on the data sample.
13. The system of claim 12 , wherein the outlier detection component classifies the data sample as an outlier data sample or an inlier data sample using the outlier detection model.
14. The system of claim 12 , wherein the data samples included in the training dataset that were projected onto the standard feature space were selected based on a determination that the machine learning model generated correct inferences on the data samples during training of the machine learning model.
15. The system of claim 12 , wherein the computer executable components further comprise:
a notification component that generates an outlier detection notification for the data sample based on classification of the data sample as an outlier data sample.
16. The system of claim 12 , wherein the computer executable components further comprise:
an inferencing component that applies the machine learning model to the data sample to generate an inference output based on classification of the data sample as an inlier data sample.
17. The system of claim 12 , wherein the computer executable components further comprise:
a reprocessing component that sends, the data sample for manual annotation and review based on classification of the data sample as an outlier data sample.
18. The system of claim 12 , wherein the standard feature space comprises a plurality of images with annotated visual features and wherein the data sample and the data samples respectively comprise images.
19. A non-transitory machine-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
projecting a data sample onto a standard feature space;
extracting a feature vector for the data sample based on the projecting using a feature extraction network trained on the standard feature space;
classifying the data sample as an outlier data sample or an inlier data sample relative to a training dataset used to train a machine learning model, wherein the classifying comprises processing the feature vector using an outlier detection model that was trained using features extracted from the training dataset based on projection of the training dataset onto the standard feature space;
generating a confidence score for the data sample using the outlier detection model, wherein the confidence score represents a measure of confidence in the machine learning model to generate an accurate inference on the data sample; and
classifying the data sample as an outlier data sample or an inlier data sample based on the confidence score.
20. The non-transitory machine-readable storage medium of claim 19 , wherein the operations further comprise:
generating an outlier detection notification for the data sample based on a first classification of the data sample as an outlier data sample; and
applying the machine learning model to the data sample to generate an inference result based on second classification of the data sample as an inlier data sample.