System and methods for performing NLP related tasks using contextualized word representations
Systems, apparatuses, and methods for representing words or phrases, and using the representation to perform NLP and NLU tasks, where these tasks include sentiment analysis, question answering, and conference resolution. Embodiments introduce a type of deep contextualized word representation that models both complex characteristics of word use, and how these uses vary across linguistic contexts. The word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. These representations can be added to existing task models and significantly improve the state of the art across challenging NLP problems, including question answering, textual entailment and sentiment analysis.
1. A method for improving the performance of a neural network used for a natural language understanding (NLU) or a natural language processing (NLP) task, comprising:
representing a natural language sequence or sequences with a bidirectional language model;
implementing the bidirectional language model in a neural network;
training the neural network in which the language model is implemented using a corpus of text;
identifying representations of the bidirectional language model in one or more layers of the trained neural network;
forming an expression based on the identified representations; and
inserting the formed expression into a layer of the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
2. The method of claim 1 , wherein the natural language understanding (NLU) or natural language processing (NLP) task is one of textual entailment, question answering, semantic role labeling, co-reference resolution, named entity extraction, or text classification.
3. The method of claim 1 , wherein the formed expression further comprises one or more factors that are determined by the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
4. The method of claim 1 , wherein the bidirectional language model includes a forward and a backward language model, with the formulation of the language model jointly maximizing the log likelihood of the forward and backward directions.
5. The method of claim 4 , wherein the bidirectional language model shares one or more weights between directions instead of using independent parameters.
6. The method of claim 1 , wherein the neural network in which the bidirectional language model is implemented is a long short-term memory (LSTM) neural network.
7. The method of claim 3 , wherein the formed expression comprises a single vector.
8. The method of claim 1 , wherein inserting the formed expression into a layer of the neural network used for the natural language understanding (NLU or natural language processing (NLP) task further comprises concatenating the formed expression with a token representation in the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
9. One or more non-transitory computer-readable media comprising a set of computer-executable instructions that when executed by one or more programmed electronic processors, cause the processors to improve the performance of a neural network used for a natural language understanding (NLU) or a natural language processing (NLP) task by:
representing a natural language sequence or sequences with a bidirectional language model;
implementing the bidirectional language model in a neural network;
training the neural network in which the language model is implemented using a corpus of text;
identifying representations of the bidirectional language model in one or more layers of the trained neural network;
forming an expression based on the identified representations; and
inserting the formed expression into a layer of the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
10. The one or more non-transitory computer-readable media of claim 9 , wherein the natural language understanding (NLU) or natural language processing (NLP) task is one of textual entailment, question answering, semantic role labeling, co-reference resolution, named entity extraction, or text classification.
11. The one or more non-transitory computer-readable media of claim 9 , wherein the bidirectional language model includes a forward and a backward language model, with the formulation of the language model jointly maximizing the log likelihood of the forward and backward directions.
12. The one or more non-transitory computer-readable media of claim 11 , wherein the bidirectional language model shares one or more weights between directions instead of using independent parameters.
13. The one or more non-transitory computer-readable media of claim 9 , wherein the neural network in which the bidirectional language model is implemented is a long short-term memory (LSTM) neural network.
14. The one or more non-transitory computer-readable media of claim 9 , wherein the formed expression further comprises one or more factors that are determined by the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
15. The one or more non-transitory computer-readable media of claim 9 , wherein inserting the formed expression into a layer of the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task further comprises concatenating the formed expression a token representation in the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
16. A system for improving the performance of a neural network used for a natural language understanding (NLU) or a natural language processing (NLP) task, comprising:
an electronic processor;
a non-transitory data storage element coupled to the processor and including a set of computer-executable instructions that, when executed by the electronic processor, cause the system to
represent a natural language sequence or sequences as a bidirectional language model;
implement the bidirectional language model in the form of a neural network;
train the neural network in which the language model is implemented using a corpus of text;
identify representations of the bidirectional language model in one or more layers of the trained neural network; and
form an expression based on the identified representations, wherein the formed expression further comprises one or more factors that are determined by the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
17. The system of claim 16 , wherein the bidirectional language model includes a forward and a backward language model, with the formulation of the language model jointly maximizing the log likelihood of the forward and backward directions.
18. The system of claim 16 , wherein the natural language of understanding (NLU) or natural language processing (NLP) task is one of textual entailment, question answering, semantic role labeling, co-reference resolution, named entity extraction, or sentiment analysis.
19. The system of claim 16 , further comprising computer-executable instructions that cause the electronic processor or a second electronic processor to insert the formed expression into a layer of the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.
20. The system of claim 19 , wherein inserting the formed expression into a layer of the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task further comprises concatenating the formed expression with a token representation in the neural network used for the natural language understanding (NLU) or natural language processing (NLP) task.