Method to decide a labeling priority to a data
According to an exemplary embodiment of the present disclosure, a computer program stored in a computer readable storage medium is disclosed. The computer program may include instructions for causing one or more processors to perform the following steps, and the steps may include: generating a data subset including at least two or more data from an unlabeled entire dataset; generating two or more training datasets from the data subset, in which the two or more training datasets includes at least one pair of sampled data subset, and in which each of at least one pair of the sampled data subset includes different validation datasets; and training a plurality of inference models, based on the two or more training datasets and corresponding labeling values.
1 . A non-transitory computer readable medium storing a computer program that comprises instructions for causing one or more processors to perform the following steps comprising:
generating a labeled data subset including at least two or more data from an unlabeled entire data set;
generating two or more training data sets from the same labeled data subset of the unlabeled entire data set, wherein each of the two or more training data sets includes at least one pair of a sampled data subset, and wherein each of at least one pair of the sampled data subset includes a different validation data set; and
training a plurality of inference models having different neural network architectures, based on the two or more training data sets and a labeling value corresponding to at least a part of the data included in the two or more training data sets by synthesizing a plurality of prediction results that are generated from the plurality of inference models by creating a plurality of prediction models based on the data in the two or more training data sets and combining the plurality of prediction models using an ensemble method to generate a final prediction model, wherein the ensemble method determines a confidence score for each prediction result, wherein the final prediction model generates a single inference result by synthesizing the plurality of prediction results using at least one of averaging, majority vote, or Bayesian inference to compute an uncertainty value, and wherein the final prediction model automatically determines, based on the confidence scores, data from the unlabeled entire data set to be labeled.
2 . The non-transitory computer readable medium according to claim 1 , wherein the generating the two or more training data sets from the data subset comprises:
generating a plurality of training data sets, each of which includes a unique validation data set, based on the two or more training data sets.
3 . The non-transitory computer readable medium according to claim 2 , wherein the generating a plurality of training data sets, each of which includes a unique validation data set, based on the two or more training data sets comprises:
dividing the two or more training data sets into N data subsets; and
allocating at least one of the N data subsets as the validation data set.
4 . The non-transitory computer readable medium according to claim 1 , wherein the plurality of inference models includes two or more network functions including different structures.
5 . The non-transitory computer readable medium according to claim 1 further comprising:
generating a plurality of inference results for random data including data in the unlabeled entire data set, based on the plurality of inference models including one or more network functions which are trained based on two or more training data sets and a corresponding labeling value; and
providing the plurality of inference results for the random data.
6 . The non-transitory computer readable medium according to claim 5 , wherein the plurality of inference results includes a classification result that each inference model infers about the random data, and a confidence score which is related to the classification result.
7 . The non-transitory computer readable medium according to claim 5 , wherein the providing the plurality of inference results for the random data comprises:
generating a single inference result for the random data based on the plurality of inference results for the random data; and
providing the single inference result.
8 . The non-transitory computer readable medium according to claim 7 , wherein the generating a single inference result for the random data based on the plurality of inference results of the random data comprises:
labeling a pseudo label to corresponding random data, when the single inference result satisfied predetermined criteria.
9 . The non-transitory computer readable medium according to claim 8 , wherein the labeling a pseudo label to corresponding random data, when the single inference result satisfied predetermined criteria comprises:
determining a classification prediction result included in the single inference result as a pseudo label for the random data, when an uncertainty value included in the single inference result is equal or less than a preset threshold.
10 . The non-transitory computer readable medium according to claim 8 , wherein the providing the single inference result comprises:
determining a labeling priority corresponding to the single inference result, against random data that has not been assigned a pseudo label.
11 . The non-transitory computer readable medium according to claim 10 , wherein the determining a labeling priority corresponding to the single inference result, against random data that has not been assigned a pseudo label:
allocating a first priority to the random data, when an uncertainty value of the random data is equal or less than a first preset threshold;
allocating a second priority to the random data, when an uncertainty value of the random data is more than the first preset threshold and equal to or less than a second preset threshold; and
allocating a third priority to the random data, when an uncertainty value of the random data is more than the second preset threshold.
12 . The non-transitory computer readable medium according to claim 5 , wherein the plurality of inference models includes two or more network functions including different structures.
13 . A computing device for determining a labeling priority, comprising: a processor; and
a storage unit storing instructions that, when retrieved and executed, cause the processor to:
generate a labeled data subset including at least two or more data from an unlabeled entire data set;
generate two or more training data sets from the same labeled data subset of the unlabeled entire data set, wherein each of the two or more training data sets includes at least one pair of a sampled data subset, and wherein each of at least one pair of the sampled data subset includes a different validation data set; and
train a plurality of inference models having different neural network architectures, based on the two or more training data sets and a labeling value corresponding to at least a part of the data included in the two or more training data sets by synthesizing a plurality of prediction results that are generated from the plurality of inference models by creating a plurality of prediction models based on the data in the two or more training data sets and combining the plurality of prediction models using an ensemble method to generate a final prediction model, wherein the ensemble method determines a confidence score for each prediction result, wherein the final prediction model generates a single inference result by synthesizing the plurality of prediction results using at least one of averaging, majority vote, or Bayesian inference to compute an uncertainty value, and wherein the final prediction model automatically determines, based on the confidence scores, data from the unlabeled entire data set to be labeled.
14 . The computing device of claim 13 , wherein the instructions further cause the processor to:
generate a plurality of inference results for random data including data in the unlabeled entire data set, based on the plurality of inference models including one or more network functions which are trained based on the two or more training data sets and a corresponding labeling value; and
provide the plurality of inference results for the random data.
15 . The computing device of claim 13 , wherein the instructions further cause the processor to generate a plurality of training data sets, each of which includes a unique validation data set, based on the two or more training data sets.
16 . The computing device of claim 13 , wherein the instructions further cause the processor to:
divide the two or more training data sets into N data subsets; and
allocate at least one of the N data subsets as the validation data set.
17 . The computing device of claim 13 , wherein the plurality of inference models includes two or more network functions including different structures.
18 . The computing device of claim 13 , wherein the instructions further cause the processor to:
generate a plurality of inference results for random data including data in the unlabeled entire data set, based on the plurality of inference models including one or more network functions which are trained based on the two or more training data sets and a corresponding labeling value;
generate a single inference result for the random data based on the plurality of inference results for the random data; and
provide the single inference result.
19 . The computing device of claim 18 , wherein the instructions further cause the processor to label a pseudo label to corresponding random data, when the single inference result satisfied predetermined criteria.
20 . The computing device of claim 19 , wherein the instructions further cause the processor to determine a labeling priority corresponding to the single inference result, against random data that has not been assigned a pseudo label.