Word embedding system
A computer-implemented method, computer program product, and computer processing system are provided for word embedding. The method includes receiving, by a processor device, a word embedding matrix. The method further includes generating, by a processor device, an average pooling vector and a max pooling vector, based on the word embedding matrix. The method also includes generating, by the processor device, a prediction by applying a Multi-Layer Perceptron (MLP) to the average pooling vector and the max pooling vector.
1. A computer-implemented method for word embedding, comprising:
receiving, by a processor device, a word embedding matrix;
generating, by a processor device, an average pooling vector and a max pooling vector, based on the word embedding matrix; and
generating, by the processor device, a prediction by applying a Multi-Layer Perceptron (MLP) to the average pooling vector and the max pooling vector.
2. The computer-implemented method of claim 1 , wherein the word embedding matrix is generated from the pre-trained word embedding for a vocabulary, and wherein the method further comprises initializing out-of-vocabulary words from a uniform distribution within a preset range.
3. The computer-implemented method of claim 1 , further comprising performing a training session for the MLP, wherein the word embeddings are directly updated during the training session.
4. The computer-implemented method of claim 1 , further comprising performing a training session for the MLP, wherein the word embeddings are kept fixed during the training session.
5. The computer-implemented method of claim 1 , further comprising concatenating the average pooling vector and the max pooling vector in preparation for use by the MLP to generate the prediction.
6. The computer-implemented method of claim 5 , further comprising forming sentence embeddings from a concatenation of the average pooling vector and the max pooling vector.
7. The computer-implemented method of claim 1 , wherein the word embedding matrix is generated from a sentence.
8. The computer-implemented method of claim 1 , wherein the word embedding matrix is generated from a document.
9. The computer-implemented method of claim 1 , further comprising concatenating feature embeddings of phrases containing multiple words.
10. The computer-implemented method of claim 9 , wherein a max pooling operation is employed on the feature maps.
11. The computer-implemented method of claim 1 , further comprising transforming, by an automatic speech recognition system, an acoustic utterance by a user into the word embedding matrix.
12. A computer program product for word embedding, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
receiving, by a processor device, a word embedding matrix;
generating, by a processor device, an average pooling vector and a max pooling vector, based on the word embedding matrix; and
generating, by the processor device, a prediction by applying a Multi-Layer Perceptron (MLP) to the average pooling vector and the max pooling vector.
13. The computer program product of claim 12 , wherein the word embedding matrix is generated based on a vocabulary, and wherein the method further comprises initializing out-of-vocabulary words from a uniform distribution within a preset range.
14. The computer program product of claim 12 , wherein the method further comprises performing a training session for the MLP, wherein the word embeddings are directly updated during the training session.
15. The computer program product of claim 12 , wherein the method further comprises performing a training session for the MLP, wherein the word embeddings are kept fixed during the training session for certain tasks.
16. The computer program product of claim 12 , further comprising concatenating the average pooling vector and the max pooling vector in preparation for use by the MLP to generate the prediction.
17. The computer program product of claim 16 , further comprising forming sentence embeddings from a concatenation of the average pooling vector and the max pooling vector.
18. The computer program product of claim 12 , further comprising concatenating embeddings of phrases containing multiple words.
19. The computer program product of claim 18 , wherein a max pooling operation is employed on the feature maps.
20. A computer processing system for word embedding, comprising:
a memory for storing program code; and
a processor device for running the program code to
receive a word embedding matrix;
generate an average pooling vector and a max pooling vector, based on the word embedding matrix; and
generate a prediction by applying a Multi-Layer Perceptron (MLP) to the average pooling vector and the max pooling vector.