Methods, systems, and media for selecting candidates for annotation for use in training classifiers

1. A method for selecting candidates for annotation for use in training classifiers, comprising:

identifying, for a trained Convolutional Neural Network (CNN), a group of candidate training samples, wherein each candidate training sample is a portion of an image, and wherein each candidate training sample includes a plurality of patches of the portion of the image;

for each candidate training sample in the group of candidate training samples:

for each patch of the plurality of patches associated with the candidate training sample, determining a plurality of probabilities, each probability being a probability that the patch corresponds to a label of a plurality of labels, wherein the plurality of probabilities are determined using the trained CNN;

identifying a subset of the patches in the plurality of patches; and

for each patch in the subset of the patches, calculating a metric that indicates at least a variance of the probabilities assigned to each patch in the subset of the patches;

selecting a subset of the candidate training samples from the group of candidate training samples by sorting the candidate samples in a descending order based on the metric, wherein the subset does not include all of the candidate training samples, wherein a higher value of the metric indicates that a candidate sample is more informative for training of the CNN, and further wherein the selecting of the subset of the candidate samples is based on the sorted candidate samples;

labeling candidate training samples in the subset of the candidate training samples by querying an external source; and

re-training the CNN using the labeled candidate training samples.

2. The method of claim 1 , further comprising identifying a group of misclassified training samples that were misclassified by the trained CNN, wherein the CNN is re-trained using both the labeled candidate training samples and the group of misclassified training samples.

3. The method of claim 1 , wherein the metric further indicates an uncertainty of the probabilities assigned to each patch in the subset of patches.

4. The method of claim 1 , wherein the subset of the candidate samples are selected using a randomization parameter.

5. The method of claim 1 , wherein each patch in the plurality of patches is generated by cropping the portion of the image corresponding to the candidate training sample.

6. A system for selecting candidates for annotation for use in training classifiers, the system comprising:

a memory; and

a hardware processor that, when executing computer-executable instructions stored in the memory, is configured to:

identify, for a trained Convolutional Neural Network (CNN), a group of candidate training samples, wherein each candidate training sample is a portion of an image, and wherein each candidate training sample includes a plurality of patches of the portion of the image;

for each candidate training sample in the group of candidate training samples:

for each patch of the plurality of patches associated with the candidate training sample, determine a plurality of probabilities, each probability being a probability that the patch corresponds to a label of a plurality of labels, wherein the plurality of probabilities are determined using the trained CNN;

identify a subset of the patches in the plurality of patches; and

for each patch in the subset of the patches, calculate a metric that indicates at least a variance of the probabilities assigned to each patch in the subset of the patches;

select a subset of the candidate training samples from the group of candidate training samples by sorting the candidate samples in a descending order based on the metric, wherein the subset does not include all of the candidate training samples, wherein a higher value of the metric indicates that a candidate sample is more informative for training of the CNN, and further wherein the selecting of the subset of the candidate samples is based on the sorted candidate samples;

label candidate training samples in the subset of the candidate training samples by querying an external source; and

re-train the CNN using the labeled candidate training samples.

7. The system of claim 6 , wherein the hardware processor is further configured to identify a group of misclassified training samples that were misclassified by the trained CNN, wherein the CNN is re-trained using both the labeled candidate training samples and the group of misclassified training samples.

8. The system of claim 6 , wherein the metric further indicates an uncertainty of the probabilities assigned to each patch in the subset of patches.

9. The system of claim 6 , wherein the subset of the candidate samples are selected using a randomization parameter.

10. The system of claim 6 , wherein each patch in the plurality of patches is generated by cropping the portion of the image corresponding to the candidate training sample.

11. Non-transitory computer-readable storage media having computer executable instructions stored thereupon that, when executed by a processor, the computer executable instructions cause the processor to perform a method for selecting candidates for annotation for use in training classifiers, the method comprising:

identifying, for a trained Convolutional Neural Network (CNN), a group of candidate training samples, wherein each candidate training sample is a portion of an image, and wherein each candidate training sample includes a plurality of patches of the portion of the image;

for each candidate training sample in the group of candidate training samples:

for each patch of the plurality of patches associated with the candidate training sample, determining a plurality of probabilities, each probability being a probability that the patch corresponds to a label of a plurality of labels, wherein the plurality of probabilities are determined using the trained CNN;

identifying a subset of the patches in the plurality of patches; and

for each patch in the subset of the patches, calculating a metric that indicates at least a variance of the probabilities assigned to each patch in the subset of the patches;

selecting a subset of the candidate training samples from the group of candidate training samples by sorting the candidate samples in a descending order based on the metric, wherein the subset does not include all of the candidate training samples, wherein a higher value of the metric indicates that a candidate sample is more informative for training of the CNN, and further wherein the selecting of the subset of the candidate samples is based on the sorted candidate samples;

labeling candidate training samples in the subset of the candidate training samples by querying an external source; and

re-training the CNN using the labeled candidate training samples.

12. The non-transitory computer-readable media of claim 11 , wherein the method further comprises identifying a group of misclassified training samples that were misclassified by the trained CNN, wherein the CNN is re-trained using both the labeled candidate training samples and the group of misclassified training samples.

13. The non-transitory computer-readable media of claim 11 , wherein the metric further indicates an uncertainty of the probabilities assigned to each patch in the subset of patches.

14. The non-transitory computer-readable media of claim 11 , wherein the subset of the candidate samples are selected using a randomization parameter.

15. The non-transitory computer-readable media of claim 11 , wherein each patch in the plurality of patches is generated by cropping the portion of the image corresponding to the candidate training sample.