Implementing a whole sentence recurrent neural network language model for natural language processing
A whole sentence recurrent neural network (RNN) language model (LM) is provided for for estimating a probability of likelihood of each whole sentence processed by natural language processing being correct. A noise contrastive estimation sampler is applied against at least one entire sentence from a corpus of multiple sentences to generate at least one incorrect sentence. The whole sentence RNN LN is trained, using the at least one entire sentence from the corpus and the at least one incorrect sentence, to distinguish the at least one entire sentence as correct. The whole sentence recurrent neural network language model is applied to estimate the probability of likelihood of each whole sentence processed by natural language processing being correct.
1. A method comprising:
providing, by a computer system, a whole sentence recurrent neural network language model for estimating a probability of likelihood of each whole sentence processed by natural language processing being correct;
applying, by the computer system, a noise contrastive estimation sampler against at least one entire sentence from a corpus of a plurality of sentences to generate at least one incorrect sentence;
training, by the computer system, the whole sentence recurrent neural network language model, using the at least one entire sentence from the corpus and the at least one incorrect sentence, to distinguish the at least one entire sentence as correct; and
applying, by the computer system, the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing being correct.
2. The method according to claim 1 , wherein applying, by the computer system, the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing further comprises:
applying, by the computer system, the whole sentence recurrent neural network language model for the natural language processing comprising one of conversational interaction, conversational telephony speech transcription, multimedia captioning, and translation.
3. The method according to claim 1 , wherein providing, by the computer system, a whole sentence recurrent neural network language model for estimating a probability of likelihood of each whole sentence processed by natural language processing further comprises:
providing, by the computer system, the whole sentence recurrent neural network language model on a recurrent neural network long short-term memory architecture.
4. The method according to claim 1 , wherein applying, by the computer system, the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing further comprises:
scoring, by the computer system, by the whole sentence recurrent neural network language model, the probability of each whole sentence directly without independently computing conditional probabilities for each separate word in each whole sentence.
5. The method according to claim 1 , wherein applying, by the computer system, a noise contrastive estimation sampler against at least one entire sentence from a corpus to generate at least one incorrect sentence further comprises:
applying, by the computer system, the noise contrastive estimation sampler against the at least one entire sentence from the corpus by performing one of a substitution, an insertion, and a deletion of one or more words in the at least one entire sentence to generate the at least one incorrect sentence.
6. The method according to claim 1 , wherein applying, by the computer system, a noise contrastive estimation sampler against at least one entire sentence from a corpus to generate at least one incorrect sentence further comprises:
randomly selecting, by the computer system, a plurality of positions in the at least one entire sentence from the corpus to introduce a substitution, an insertion, and a deletion of one or more words in the at least one entire sentence to generate the at least one incorrect sentence.
7. The method according to claim 1 , wherein applying, by the computer system, the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing being correct further comprises:
selecting, by the computer system, a selection of the plurality of sentences from the corpus;
applying, by the computer system, the noise contrastive estimation sampler against each sentence in the selection of the plurality of sentences to generate a plurality of imposter sentences;
applying, by the computer system, each separate set of each sentence in the selection of the plurality of sentences and a selection of imposter sentences of the plurality of imposter sentences generated for each sentence to the whole sentence recurrent neural network language model;
generating, by the computer system, through the whole sentence recurrent neural network language model, a first score for each sentence and at least one additional score for the selection of imposter sentences;
applying, by the computer system, a linear boundary to classify the first score and the additional score in one of two classes in a linear space, wherein the two classes represent an incorrect sentence and a correct sentence; and
evaluating, by the computer system, an accuracy of the natural language processing system in performing sequential classification tasks based on an accuracy of the classifications of the first score in the class of the correct sentence and at least one additional score as an incorrect sentence.
8. The method according to claim 1 , wherein training, by the computer system, the whole sentence recurrent neural network language model, using the at least one entire sentence from the corpus and the at least one incorrect sentence, to distinguish the at least one entire sentence as correct further comprises:
applying, by the computer system, each of the at least one entire sentence from the corpus and the at least one incorrect sentence to at least one recurrent neural network layer comprising a plurality of long short-term memory for holding data for an arbitrary period of time;
pushing, by the computer system, an output from each of the plurality of long short-term memory to a neural network scorer for each of the at least one entire sentence and the at least one incorrect sentence;
generating, by the neural network scorer, a separate output score assigned by the at least one recurrent neural network layer for each of the at least one entire sentence and the at least one incorrect sentence representing an unnormalized probability of each sentence; and
evaluating, by a neural network layer receiving output from the whole sentence recurrent neural network language model, each separate output score as an output of a digital 1 if the output score is a probability indicating the entire sentence is correct and an output of a digital 0 if the output score is a probability indicating the entire sentence is not correct.
9. A computer system comprising one or more processors, one or more computer-readable memories, one or more computer-readable storage devices, and program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the stored program instructions comprising:
program instructions to provide a whole sentence recurrent neural network language model for estimating a probability of likelihood of each whole sentence processed by natural language processing being correct;
program instructions to apply a noise contrastive estimation sampler against at least one entire sentence from a corpus of a plurality of sentences to generate at least one incorrect sentence;
program instructions to train the whole sentence recurrent neural network language model, using the at least one entire sentence from the corpus and the at least one incorrect sentence, to distinguish the at least one entire sentence as correct; and
program instructions to apply the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing being correct.
10. The computer system according to claim 9 , wherein the program instructions to apply the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing further comprise:
program instructions to apply the whole sentence recurrent neural network language model for the natural language processing comprising one of conversational interaction, conversational telephony speech transcription, multimedia captioning, and translation.
11. The computer system according to claim 9 , wherein the program instructions to provide a whole sentence recurrent neural network language model for estimating a probability of likelihood of each whole sentence processed by natural language processing further comprise:
program instructions to provide the whole sentence recurrent neural network language model on a recurrent neural network long short-term memory architecture.
12. The computer system according to claim 9 , wherein the program instructions to apply the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing further comprise:
program instructions to score, by the whole sentence recurrent neural network language model, the probability of each whole sentence directly without independently computing conditional probabilities for each separate word in each whole sentence.
13. The computer system according to claim 9 , the program instructions to apply a noise contrastive estimation sampler against at least one entire sentence from a corpus to generate at least one incorrect sentence further comprise:
program instructions to apply the noise contrastive estimation sampler against the at least one entire sentence from the corpus by performing one of a substitution, an insertion, and a deletion of one or more words in the at least one entire sentence to generate the at least one incorrect sentence.
14. The computer system according to claim 9 , wherein the program instructions to apply a noise contrastive estimation sampler against at least one entire sentence from a corpus to generate at least one incorrect sentence further comprise:
program instructions to randomly select a plurality of positions in the at least one entire sentence from the corpus to introduce a substitution, an insertion, and a deletion of one or more words in the at least one entire sentence to generate the at least one incorrect sentence.
15. The computer system according to claim 9 , wherein the program instructions to apply the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing being correct further comprise:
program instructions to select a selection of the plurality of sentences from the corpus;
program instructions to apply the noise contrastive estimation sampler against each sentence in the selection of the plurality of sentences to generate a plurality of imposter sentences;
program instructions to apply each separate set of each sentence in the selection of the plurality of sentences and a selection of imposter sentences of the plurality of imposter sentences generated for each sentence to the whole sentence recurrent neural network language model;
program instructions to generate, by the whole sentence recurrent neural network language model, a first score for each sentence and at least one additional score for the selection of imposter sentences;
program instructions to apply a linear boundary to classify the first score and the additional score in one of two classes in a linear space, wherein the two classes represent an incorrect sentence and a correct sentence; and
program instructions to evaluate an accuracy of the natural language processing system in performing sequential classification tasks based on an accuracy of the classifications of the first score in the class of the correct sentence and at least one additional score as an incorrect sentence.
16. The computer system according to claim 9 , wherein the program instructions to train the whole sentence recurrent neural network language model, using the at least one entire sentence from the corpus and the at least one incorrect sentence, to distinguish the at least one entire sentence as correct further comprise:
program instructions to apply each of the at least one entire sentence from the corpus and the at least one incorrect sentence to at least one recurrent neural network layer comprising a plurality of long short-term memory for holding data for an arbitrary period of time;
program instructions to push an output from each of the plurality of long short-term memory to a neural network scorer for each of the at least one entire sentence and the at least one incorrect sentence;
program instructions to generate, by the neural network scorer, a separate output score assigned by the at least one recurrent neural network layer for each of the at least one entire sentence and the at least one incorrect sentence representing an unnormalized probability of each sentence; and
program instructions to evaluate, by a neural network layer receiving output from the whole sentence recurrent neural network language model, each separate output score as an output of a digital 1 if the output score is a probability indicating the entire sentence is correct and an output of a digital 0 if the output score is a probability indicating the entire sentence is not correct.
17. A computer program product comprises a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a computer to cause the computer to:
provide, by a computer, a whole sentence recurrent neural network language model for estimating a probability of likelihood of each whole sentence processed by natural language processing being correct;
apply, by the computer, a noise contrastive estimation sampler against at least one entire sentence from a corpus of a plurality of sentences to generate at least one incorrect sentence;
train, by the computer, the whole sentence recurrent neural network language model, using the at least one entire sentence from the corpus and the at least one incorrect sentence, to distinguish the at least one entire sentence as correct; and
apply, by the computer, the whole sentence recurrent neural network language model to estimate the probability of likelihood of each whole sentence processed by natural language processing being correct.
18. The computer program product according to claim 17 , further comprising the program instructions executable by a computer to cause the computer to:
apply, by the computer, the whole sentence recurrent neural network language model for the natural language processing comprising one of conversational interaction, conversational telephony speech transcription, multimedia captioning, and translation.
19. The computer program product according to claim 17 , further comprising the program instructions executable by a computer to cause the computer to:
provide, by the computer, the whole sentence recurrent neural network language model on a recurrent neural network long short-term memory architecture.
20. The computer program product according to claim 17 , further comprising the program instructions executable by a computer to cause the computer to:
score, by the computer, by the whole sentence recurrent neural network language model, the probability of each whole sentence directly without independently computing conditional probabilities for each separate word in each whole sentence.