Human-assisted training of automated classifiers
Many computing scenarios involve the classification of content items within one or more categories. The content item set may be too large for humans to classify, but an automated classifier (e.g., an artificial neural network) may not be able to classify all content items with acceptable accuracy. Instead, the automated classifier may calculate a classification confidence while classifying respective content items. Content items having a low classification confidence may be sent to a human classifier, and may be added, along with the categories identified by the human classifier, to a training set. The automated classifier may then be retrained using the training set, thereby incrementally improving the classification confidence of the automated classifier while conserving the involvement of human classifiers. Additionally, human classifiers may be rewarded for classifying the content items, and the costs of such rewards may be considered while selecting content items for the training set.
1. A method of training an automated classifier to identify topical categories of content items of a content set using a device comprising a processor and having access to at least one human classifier, the method comprising:
executing on the processor instructions configured to:
for respective content items of the content set:
using the automated classifier, identify at least one topical category for the content item with a classification confidence;
for respective content items having a classification confidence for a selected topical category that is below a classification confidence threshold:
send the content item to a human classifier including the selected topical category into which the content item has been classified, and
upon receiving from the human classifier at least one user-selected topical category associated with the content item, store the content item in a training set associated with the at least one user-selected topical category; and
train the automated classifier using the training set.
2. The method of claim 1 , the method comprising: from the content items having a classification confidence below the classification confidence threshold, selecting for sending to human classifiers the content items having the lowest classification confidences identified by the automated classifier.
3. The method of claim 1 :
the classification confidence computed for respective topical categories;
the instructions further configured to, for respective topical categories, calculate an aggregate classification confidence of the content items associated with the topical category; and
sending content items to the human classifier comprising: for a selected topical categories having an aggregate classification confidence below a classification confidence threshold, sending at least one of the content items classified with the selected topical category to the human classifier.
4. The method of claim 1 :
the automated classifier configured to associate respective content items with respective topical categories having a classification degree; and
the instructions configured to, upon receiving from at least one human classifier at least one topical category associated with the content item, associate the content item in the training set with at least one topical category having a classification degree calculated proportionally to the at least one human classifier associating the content item with the topical category.
5. The method of claim 1 :
the device having access to at least two human classifiers; and
the instructions configured to:
send content items having a classification confidence below the classification confidence threshold to at least two human classifiers; and
upon receiving from the at least two human classifier at least one topical category associated with the content item, associate the content item in the training set with at least one topical category associated with the content item by the at least two human classifiers.
6. The method of claim 1 , the instructions configured to, upon receiving the at least one topical category associated with the content item from the human classifier, deliver a reward to the human classifier.
7. The method of claim 6 , the training set comprising:
a first content item associated with a first reward, and
a second content item associated with a second reward that is different from the first reward.
8. The method of claim 7 :
respective human classifiers having a human classifier rating; and
the reward delivered to a human classifier calculated proportionally to the human classifier rating of the human classifier.
9. The method of claim 7 , the reward associated with a content item calculated proportionally to a training value of the content item in the training set.
10. The method of claim 7 , the reward calculated inversely proportionally to the classification confidence of the content item identified by the automated classifier.
11. The method of claim 7 , the reward associated with a content item calculated proportionally to a classification complexity of the content item.
12. The method of claim 7 , selecting the content items comprising: from the content items having a classification confidence below the classification confidence threshold, selecting the content items associated with the smallest rewards.
13. The method of claim 6 :
the device defining a reward limit of rewards provided to human classifiers; and
the method comprising: selecting for sending to human classifiers the content items associated with rewards having a reward total not exceeding the reward limit.
14. The method of claim 1 , the instructions configured to initialize the automated classifier to select at least one topical category for respective content items.
15. The method of claim 1 :
the instructions configured to initialize the training set;
identifying the content item comprising: upon identifying the content item, adding the content item to the training set; and
sending the content item to a human classifier comprising: upon the training set reaching a training set size, sending respective content items of the training set to a human classifier.
16. The method of claim 1 , training the automated classifier comprising:
training the automated classifier using the training set; and
after training the automated classifier:
calculating a classification confidence for the automated classifier; and
while the classification confidence is below the classification confidence threshold:
identifying additional content items having a classification confidence below the classification confidence threshold, and
retraining the training set using the additional content items.
17. The method of claim 16 , training the automated classifier comprising: after training the automated classifier using the training set, reinitializing the training set.
18. The method of claim 1 :
at least one content item having a classification confidence below a classification confidence threshold for at least two topical categories; and
sending the content item to a human classifier further comprising:
informing the human classifier of the at least two topical categories with which the content item has been classified; and
requesting the human classifier to select at least one of the topical categories with which the content item is to be classified.
19. A system configured to train an automated classifier to identify topical categories of content items of a content set using a device comprising a memory and a processor and having access to at least one human classifier, the system comprising:
a content item selecting component comprising instructions stored in the memory that, when executed on the processor, cause the device to, for respective content items of the content set, using the automated classifier, identify at least one topical category for the content item with a classification confidence;
a training set generating component comprising instructions stored in the memory that, when executed on the processor, cause the device to:
for respective topical categories, calculate an aggregate classification confidence of the content items associated with the topical category;
for a selected topical category having an aggregate classification confidence below a classification confidence threshold:
send at least one of the content items classified with the selected topical category to the human classifier; and
upon receiving from the human classifier at least one human-classified topical category associated with the content item, store the content item in a training set associated with the at least one human-classified topical category; and
an automated classifier training component comprising instructions stored in the memory that, when executed on the processor, cause the device to train the automated classifier using the training set.
20. A computer-readable storage device comprising instructions that, when executed by a processor of a device comprising an automated classifier and having access to at least one human classifier, identify topical categories of content items of a content set by:
for respective content items of the content set:
using the automated classifier, identifying at least one topical category for the content item with a classification confidence;
for respective content items having a classification confidence for a selected topical category that is below a classification confidence threshold:
sending the content item to a human classifier including the selected topical category into which the content item has been classified, and
upon receiving from the human classifier at least one user-selected topical category associated with the content item, storing the content item in a training set associated with the at least one user-selected topical category; and
training the automated classifier using the training set.