Deep neural network-based decision network
The technology disclosed proposes using a combination of computationally cheap, less-accurate bag of words (BoW) model and computationally expensive, more-accurate long short-term memory (LSTM) model to perform natural processing tasks such as sentiment analysis. The use of cheap, less-accurate BoW model is referred to herein as “skimming”. The use of expensive, more-accurate LSTM model is referred to herein as “reading”. The technology disclosed presents a probability-based guider (PBG). PBG combines the use of BoW model and the LSTM model. PBG uses a probability thresholding strategy to determine, based on the results of the BoW model, whether to invoke the LSTM model for reliably classifying a sentence as positive or negative. The technology disclosed also presents a deep neural network-based decision network (DDN) that is trained to learn the relationship between the BoW model and the LSTM model and to invoke only one of the two models.
1. A system for performing a selection task using a decision neural network-based classifier, comprising: a memory storing a plurality of processor-executable instructions for training and operating the decision neural network-based classifier; and a processor that reads the plurality of processor-executable instructions from the memory to perform operations comprising: constructing a decision training set for the decision neural network-based classifier by: identifying, based on a confusion matrix generated from performance of a non-recurrent network-based classifier and a recurrent neural network-based classifier, a first subset of inputs inferred by the recurrent neural network-based classifier and a second subset of inputs comprising inputs not in the first subset, and including, in the decision training set, the first subset of inputs labeled with a first model class label identifying the recurrent neural network-based classifier and the second subset of inputs labeled with a second model class label identifying the non-recurrent neural network-based classifier; generating, by the decision neural network based classifier, a plurality of selection outputs in response to inputs from the decision training set; computing a training objective by comparing the plurality of selection outputs with labels from the first subset and the second subset; and updating the decision neural network based classifier based on the training objective wherein the operations further comprise: training the non-recurrent and recurrent neural network-based classifiers to perform the machine classification task using a training set, the training set comprising training inputs annotated with task class labels defined for the machine classification task; using the trained non-recurrent and recurrent neural network-based classifiers to perform the machine classification task on a validation set, the validation set comprising validation inputs annotated with the task class labels; and
training the decision neural network-based classifier using the decision training set to output probabilities for the first and second model class labels on an input-by-input basis, the output probabilities specifying respective likelihoods of selecting the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier.
2. The system of claim 1 , wherein the operations further comprise: at an inference stage, selecting, by the decision neural network-based classifier, on an input-by-input basis, between the non-recurrent neural network-based classifier and the recurrent neural network-based classifier to perform a machine classification task; and based on output probabilities of the trained decision neural network-based classifier, performing the machine classification task on the given input using either the trained recurrent neural network-based classifier or the trained non-recurrent neural network-based classifier.
3. The system of claim 2 , wherein the operation further comprise: selecting the trained recurrent neural network-based classifier when an output probability associated with the first model class label is higher than that of the second model class label; or selecting the trained non-recurrent neural network-based classifier when an output probability associated with the second model class label is higher than that of the first model class label.
4. The system of claim 1 , wherein the non-recurrent network-based classifier and the recurrent neural network-based classifier are separately trained on the decision training set.
5. The system of claim 1 , wherein the decision training set comprises a plurality of sentences annotated with a sentiment label.
6. The system of claim 1 , wherein the confusion matrix identifies at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier; and validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.
7. The system of claim 1 , wherein the first subset includes validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.
8. The system of claim 1 , wherein the second subset includes at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; and validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier.
9. The system of claim 1 , wherein an operation of updating the decision neural network based classifier comprises: backpropagating gradients only for fully-connected layers and a new classification layer while keeping weights of the trained non-recurrent neural network-based classifier fixed.
10. A method for performing a selection task using a decision neural network-based classifier, the method comprising: constructing a decision training set for the decision neural network-based classifier by: identifying, based on a confusion matrix generated from performance of a non-recurrent network-based classifier and a recurrent neural network-based classifier, a first subset of inputs inferred by the recurrent neural network-based classifier and a second subset of inputs comprising inputs not in the first subset, and including, in the decision training set, the first subset of inputs labeled with a first model class label identifying the recurrent neural network-based classifier and the second subset of inputs labeled with a second model class label identifying the non-recurrent neural network-based classifier; generating, by the decision neural network based classifier, a plurality of selection outputs in response to inputs from the decision training set; computing a training objective by comparing the plurality of selection outputs with labels from the first subset and the second subset; and updating the decision neural network based classifier based on the training objective; training the non-recurrent and recurrent neural network-based classifiers to perform the machine classification task using a training set, the training set comprising training inputs annotated with task class labels defined for the machine classification task; using the trained non-recurrent and recurrent neural network-based classifiers to perform the machine classification task on a validation set, the validation set comprising validation inputs annotated with the task class labels; and training the decision neural network-based classifier using the decision training set to output probabilities for the first and second model class labels on an input-by-input basis, the output probabilities specifying respective likelihoods of selecting the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier.
11. The method of claim 10 , further comprising: at an inference stage, selecting, by the decision neural network-based classifier, on an input-by-input basis, between the non-recurrent neural network-based classifier and the recurrent neural network-based classifier to perform a machine classification task; and based on output probabilities of the trained decision neural network-based classifier, performing the machine classification task on the given input using either the trained recurrent neural network-based classifier or the trained non-recurrent neural network-based classifier.
12. The method of claim 11 , further comprising: selecting the trained recurrent neural network-based classifier when an output probability associated with the first model class label is higher than that of the second model class label; or selecting the trained non-recurrent neural network-based classifier when an output probability associated with the second model class label is higher than that of the first model class label.
13. The method of claim 10 , wherein the non-recurrent network-based classifier and the recurrent neural network-based classifier are separately trained on the decision training set.
14. The method of claim 10 , wherein the decision training set comprises a plurality of sentences annotated with a sentiment label.
15. The method of claim 10 , wherein the confusion matrix identifies at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.
16. The method of claim 10 , wherein the first subset includes validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier, and the second subset includes at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; and validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier.
17. The method of claim 10 , wherein updating the decision neural network based classifier comprises: backpropagating gradients only for fully-connected layers and a new classification layer while keeping weights of the trained non-recurrent neural network-based classifier fixed.
18. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for performing a selection task using a decision neural network-based classifier, the instructions being executed by a processor to perform operations comprising: constructing a decision training set for the decision neural network-based classifier by: identifying, based on a confusion matrix generated from performance of a non-recurrent network-based classifier and a recurrent neural network-based classifier, a first subset of inputs inferred by the recurrent neural network-based classifier and a second subset of inputs comprising inputs not in the first subset, and including, in the decision training set, the first subset of inputs labeled with a first model class label identifying the recurrent neural network-based classifier and the second subset of inputs labeled with a second model class label identifying the non-recurrent neural network-based classifier; generating, by the decision neural network based classifier, a plurality of selection outputs in response to inputs from the decision training set; computing a training objective by comparing the plurality of selection outputs with labels from the first subset and the second subset; and updating the decision neural network based classifier based on the training objective; training the non-recurrent and recurrent neural network-based classifiers to perform the machine classification task using a training set, the training set comprising training inputs annotated with task class labels defined for the machine classification task; using the trained non-recurrent and recurrent neural network-based classifiers to perform the machine classification task on a validation set, the validation set comprising validation inputs annotated with the task class labels; and training the decision neural network-based classifier using the decision training set to output probabilities for the first and second model class labels on an input-by-input basis, the output probabilities specifying respective likelihoods of selecting the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier.