Using multiple trained models to reduce data labeling efforts
A method of labeling a dataset includes inputting a testing set comprising a plurality of input data samples into a plurality of pre-trained machine learning models to generate a set of embeddings output by the plurality of pre-trained machine learning models. The method further includes performing an iterative cluster labeling algorithm that includes generating a plurality of clusterings from the set of embeddings, analyzing the plurality of clusterings to identify a target embedding with a highest duster quality, analyzing the target embedding to determine a compactness for each of the plurality of clusterings of the target embedding, and identifying a target cluster among the plurality of clusterings of the target embedding based on the compactness. The method further includes assigning pseudo-labels to the subset of the plurality of input data samples that are members of the target duster.
1. A method, comprising:
inputting a testing set comprising a plurality of input data samples into each one of a plurality of pre-trained machine learning models to generate a set of embeddings output by the plurality of pre-trained machine learning models; and
performing, by a processing device, N iterations of a cluster labeling algorithm, wherein each iteration comprises:
generating a plurality of clusterings from the set of embeddings;
analyzing the plurality of clusterings to identify a target embedding with a highest cluster quality among the set of embeddings;
analyzing the target embedding to determine a compactness for each of the plurality of clusterings of the target embedding;
identifying a target cluster among the plurality of clusterings of the target embedding based on the compactness, wherein a subset of the plurality of input data samples are members of the target cluster; and
assigning pseudo-labels to the subset of the plurality of input data samples and removing the subset from the testing set for all embeddings.
2. The method of claim 1 , wherein identifying the target cluster based on the compactness comprises identifying the target cluster based on the target cluster having a largest compactness among the plurality of clusterings of the target embedding.
3. The method of claim 1 , wherein identifying the target cluster based on the compactness comprises identifying the target cluster based on the target cluster having a lowest compactness among the plurality of clusterings of the target embedding.
4. The method of claim 3 , wherein the N iterations of the cluster labeling algorithm result in a first set of N class clusters, the method further comprising:
performing an additional N iterations of the cluster labeling algorithm over the set of embeddings to generate a second set of N class clusters, wherein, at each iteration, identifying the target cluster based on the compactness comprises identifying the target cluster based on the target cluster having a largest compactness among the plurality of clusterings of the target embedding; and
deploying the first set of N class clusters or the second set of N class clusters based on which set exhibits better total compactness.
5. The method of claim 1 , wherein the set of embeddings output by the plurality of pre-trained machine learning models are generated from a plurality of layer depths within the plurality of pre-trained machine learning models.
6. The method of claim 1 , wherein each of the plurality of pre-trained machine learning models is configured to perform a machine learning task comprising at least one of: an image classification task, a video classification task, or an audio classification task.
7. The method of claim 1 , further comprising:
receiving a human label corresponding to a first input data sample of the plurality of input data samples; and
assigning the human label to a second input data sample of the plurality of input data samples based on the second input data sample having a same pseudo-label as the first input data sample.
8. A system comprising:
a memory to store:
a testing set comprising a plurality of input data samples; and
a plurality of pre-trained machine learning models that are related to a machine learning task; and
a processing device, operatively coupled to the memory, to:
input the testing set comprising the plurality of input data samples into each of the plurality of pre-trained machine learning models to generate a set of embeddings output by the plurality of pre-trained machine learning models; and
perform N iterations of a cluster labeling algorithm, wherein at each iteration the processing device is to:
generate a plurality of clusterings from the set of embeddings;
analyze the plurality of clusterings to identify a target embedding with a highest cluster quality among the set of embeddings;
analyze the target embedding to determine a compactness for each of the plurality of clusterings of the target embedding;
identify a target duster among the plurality of clusterings of the target embedding based on the compactness, wherein a subset of the plurality of input data samples are members of the target cluster; and
assign pseudo-labels to the subset of the plurality of input data samples and remove the subset from the testing set for all embeddings.
9. The system of claim 8 , wherein to identify the target cluster based on the compactness, the processing device is to identify the target cluster based on the target cluster having a largest compactness among the plurality of clusterings of the target embedding.
10. The system of claim 8 , wherein to identify the target cluster based on the compactness, the processing device is to identify the target cluster based on the target cluster having a lowest compactness among the plurality of clusterings of the target embedding.
11. The system of claim 10 , wherein the N iterations of the cluster labeling algorithm result in a first set of N class clusters, and wherein the processing device is further to:
perform an additional N iterations of the cluster labeling algorithm over the set of embeddings to generate a second set of N class clusters, wherein, at each iteration, the processing device is to identify the target cluster based on the target cluster having a largest compactness among the plurality of clusterings of the target embedding; and
deploy the first set of N class clusters or the second set of N class clusters based on which set exhibits better total compactness.
12. The system of claim 8 , wherein the set of embeddings output by the plurality of pre-trained machine learning models are generated from a plurality of layer depths within the plurality of pre-trained machine learning models.
13. The system of claim 8 , wherein each of the plurality of pre-trained machine learning models is configured to perform a machine learning task comprising at least one of: an image classification task, a video classification task, or an audio classification task.
14. The system of claim 8 , wherein the processing device is further to:
receive a human label corresponding to a first input data sample of the plurality of input data samples; and
assign the human label to a second input data sample of the plurality of input data samples based on the second input data sample having a same pseudo-label as the first input data sample.
15. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
input a testing set comprising a plurality of input data samples into each one of a plurality of pre-trained machine learning models to generate a set of embeddings output by the plurality of pre-trained machine learning models; and
perform N iterations of a cluster labeling algorithm, wherein at each iteration the processing device is to:
generate a plurality of clusterings from the set of embeddings;
analyze the plurality of clusterings to identify a target embedding with a highest cluster quality among the set of embeddings;
analyze the target embedding to determine a compactness for each of the plurality of clusterings of the target embedding;
identify a target cluster among the plurality of clusterings of the target embedding based on the compactness, wherein a subset of the plurality of input data samples are members of the target cluster; and
assign pseudo-labels to the subset of the plurality of input data samples and remove the subset from the testing set for all embeddings.
16. The non-transitory computer-readable storage medium of claim 15 , wherein to identify the target cluster based on the compactness, the instructions cause the processing device to identify the target cluster based on the target cluster having a largest compactness among the plurality of clusterings of the target embedding.
17. The non-transitory computer-readable storage medium of claim 15 , wherein to identify the target cluster based on the compactness, the instructions cause the processing device to identify the target cluster based on the target cluster having a lowest compactness among the plurality of clusterings of the target embedding.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the N iterations of the cluster labeling algorithm result in a first set of N class clusters, and wherein the instructions further cause the processing device to:
perform an additional N iterations of the cluster labeling algorithm over the set of embeddings to generate a second set of N class clusters, wherein, at each iteration, the processing device is to identify the target cluster based on the target cluster having a largest compactness among the plurality of clusterings of the target embedding; and
deploy the first set of N class clusters or the second set of N class clusters based on which set exhibits better total compactness.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the set of embeddings output by the plurality of pre-trained machine learning models are generated from a plurality of layer depths within the plurality of pre-trained machine learning models.
20. The non-transitory computer-readable storage medium of claim 15 , wherein each of the plurality of pre-trained machine learning models is configured to perform a machine learning task comprising at least one of: an image classification task, a video classification task, or an audio classification task.