Real time learning of text classification models for fast and efficient labeling of training data and customization
Techniques for real-time generation and customization of text classification models. An initial dataset of input text samples are manually assigned labels, and the labeled input text samples are tokenized and provided as training data to train machine learning classifiers for various classes or categories of the input text samples. As the machine learning classifiers train with the training data, feedback in the form of suggestions (or predictions) are provided in real time by the text classification models regarding which label(s) to assign to any input text sample(s) currently in the training data or any new input text sample(s) further provided as training data for the respective machine learning classifiers. The suggested (or predicted) label(s) can be manually assigned to the input text sample(s), if deemed appropriate, and the newly labeled input text sample(s) can be provided to supplement the existing training data for the respective machine learning classifiers.
1. A computer-implemented method of training a plurality of text classification models, comprising:
predefining multiple labels corresponding to various classes of input text samples for an initial dataset of the input text samples;
manually assigning the predefined labels to selected ones of the initial dataset of the input text samples to indicate that the input text samples belong in particular ones of the various classes;
once a predefined label has been manually assigned to each of the initial dataset of the input text samples, providing the initial dataset of the input text samples as training data to train a plurality of machine learning classifiers for generating the plurality of text classification models, respectively, the plurality of text classification models corresponding to the various classes, respectively;
generating the plurality of text classification models taking into account individual keywords or pairs of keywords that occur in a predetermined threshold number of positive examples of the input text samples for a particular class, but do not occur in more than a specified number of negative examples of the input text samples for the particular class;
while the plurality of machine learning classifiers train with the training data, providing label suggestions in real-time by the plurality of text classification models regarding what predefined labels to assign to any of the input text samples in the training data;
manually assigning the predefined labels to at least some of the input text samples in accordance with the label suggestions from the plurality of text classification models; and
providing the at least some of the input text samples as supplements to the training data for the plurality of machine learning classifiers.
2. The computer-implemented method of claim 1 further comprising:
once a predefined label has been manually assigned to each of the initial dataset of the input text samples, tokenizing the initial dataset of the input text samples, and
wherein the providing of the initial dataset of the input text samples includes providing the tokenized initial dataset of the input text samples as training data to train the plurality of machine learning classifiers.
3. The computer-implemented method of claim 1 further comprising:
determining whether or not the plurality of machine learning classifiers require additional training based at least on statistics pertaining to the respective text classification models.
4. The computer-implemented method of claim 1 further comprising:
having determined that the plurality of machine learning classifiers do not require additional training, providing further label suggestions in real-time by the plurality of text classification models for subsequent input text samples for the various classes of the input text samples.
5. The computer-implemented method of claim 1 wherein the generating of the plurality of text classification models includes weighting each of the individual keywords by a weighting parameter, |N/W|, “N” corresponding to a number of occurrences of the individual keyword in a respective input text sample, and “W” corresponding to a total number of words in the respective input text sample.
6. The computer-implemented method of claim 1 wherein the generating of the plurality of text classification models includes taking into account the individual keywords that occur in purportedly positive examples of the input text samples for the particular class, but indicate that the purportedly positive examples are more appropriately regarded as negative examples of the input text samples for the particular class.
7. The computer-implemented method of claim 1 wherein the generating of the plurality of text classification models includes taking into account an order of the pairs of keywords in the input text samples while making determinations as to whether the input text samples are positive examples or negative examples for particular classes of the input text samples.
8. The computer-implemented method of claim 1 further comprising:
once a predefined label has been manually assigned to each of the initial dataset of the input text samples, vectorizing the initial dataset of the input text samples, and
wherein the providing of the initial dataset of the input text samples includes providing the vectorized initial dataset of the input text samples as training data to train the plurality of machine learning classifiers.
9. A computer-implemented method of training a plurality of text classification models, comprising:
predefining multiple labels corresponding to various classes of input text samples for an initial dataset of the input text samples;
manually assigning the predefined labels to selected ones of the initial dataset of the input text samples to indicate that the input text samples belong in particular ones of the various classes;
once a predefined label has been manually assigned to each of the initial dataset of the input text samples, providing the initial dataset of the input text samples as training data to train a plurality of machine learning classifiers for generating the plurality of text classification models, respectively, the plurality of text classification models corresponding to the various classes, respectively;
generating the plurality of text classification models taking into account a covariance of attributes that can be expressed as scalars or Boolean dimensions, the attributes including one or more of individual keywords, weighted keywords, and dimensions of a vectorized keyword in the initial dataset of the input text samples;
while the plurality of machine learning classifiers train with the training data, providing label suggestions in real-time by the plurality of text classification models regarding what predefined labels to assign to any of the input text samples in the training data;
manually assigning the predefined labels to at least some of the input text samples in accordance with the label suggestions from the plurality of text classification models; and
providing the at least some of the input text samples as supplements to the training data for the plurality of machine learning classifiers.
10. A system for training a plurality of text classification models, comprising:
a server including at least one processor operative to execute at least one program out of at least one memory:
to receive an initial dataset of input text samples with predefined labels manually assigned thereto to indicate that the input text samples belong in particular ones of various classes;
to provide the initial dataset of the input text samples as training data to train a plurality of machine learning classifiers for generating the plurality of text classification models, respectively, the plurality of text classification models corresponding to the various classes, respectively;
to generate the plurality of text classification models taking into account individual keywords or pairs of keywords that occur in a predetermined threshold number of positive examples of the input text samples for a particular class, but do not occur in more than a specified number of negative examples of the input text samples for the particular class;
while the plurality of machine learning classifiers train with the training data, to provide, through the plurality of text classification models, label suggestions in real-time regarding what predefined labels to assign to any of the input text samples in the training data;
to receive at least some of the input text samples with the predefined labels manually assigned thereto in accordance with the label suggestions; and
to provide the at least some of the input text samples as supplements to the training data for the plurality of machine learning classifiers.
11. The system of claim 10 wherein the at least one processor is further operative to execute the at least one program out of the at least one memory:
to tokenize the initial dataset of the input text samples; and
to provide the tokenized initial dataset of the input text samples as training data to train the plurality of machine learning classifiers.
12. The system of claim 10 wherein the at least one processor is further operative to execute the at least one program out of the at least one memory:
to determine whether or not the plurality of machine learning classifiers require additional training based at least on statistics pertaining to the respective text classification models.
13. The system of claim 10 wherein the at least one processor is further operative to execute the at least one program out of the at least one memory:
having determined that the plurality of machine learning classifiers do not require additional training, to provide, through the plurality of text classification models, further label suggestions in real-time for subsequent input text samples for the various classes of the input text samples.
14. The system of claim 10 wherein the server including the at least one processor and the at least one memory is configured to implement one or more of an online meeting system, a desktop sharing system, and a video conferencing system.
15. A method comprising:
receiving an initial dataset of input text samples with predefined labels manually assigned thereto to indicate that the input text samples belong in particular ones of various classes;
providing the initial dataset of the input text samples as training data to train a plurality of machine learning classifiers for generating a plurality of text classification models, respectively, the plurality of text classification models corresponding to the various classes, respectively;
while the plurality of machine learning classifiers train with the training data, providing, through the plurality of text classification models, label suggestions in real-time regarding what predefined labels to assign to any of the input text samples in the training data;
receiving at least some of the input text samples with the predefined labels manually assigned thereto in accordance with the label suggestions; and
providing the at least some of the input text samples as supplements to the training data for the plurality of machine learning classifiers.
16. The method of claim 15 further comprising:
tokenizing the initial dataset of the input text samples; and
providing the tokenized initial dataset of the input text samples as training data to train the plurality of machine learning classifiers.
17. The method of claim 15 further comprising:
determining whether or not the plurality of machine learning classifiers require additional training based at least on statistics pertaining to the respective text classification models.
18. The method of claim 15 further comprising:
having determined that the plurality of machine learning classifiers do not require additional training, providing, through the plurality of text classification models, further label suggestions in real-time for subsequent input text samples for the various classes of the input text samples.
19. The method of claim 15 further comprising:
while the plurality of machine learning classifiers train with the training data, providing, through the plurality of text classification models, label suggestions in real-time for at least some of the input text samples that would result in high information gain for training the plurality of machine learning classifiers.
20. The method of claim 15 further comprising:
generating the plurality of text classification models taking into account individual keywords or pairs of keywords that occur in a predetermined threshold number of positive examples of the input text samples for a particular class, but do not occur in more than a specified number of negative examples of the input text samples for the particular class.