Fusion of word embeddings and word scores for text classification
Techniques disclosed herein relate generally to text classification and include techniques for fusing word embeddings with word scores for text classification. In one particular aspect, a method for text classification is provided that includes obtaining an embedding vector for a textual unit, based on a plurality of word embedding vectors and a plurality of word scores. The plurality of word embedding vectors includes a corresponding word embedding vector for each of a plurality of words of the textual unit, and the plurality of word scores includes a corresponding word score for each of the plurality of words of the textual unit. The method also includes passing the embedding vector for the textual unit through at least one feed-forward layer to obtain a final layer output, and performing a classification on the final layer output.
1 . A method of classifying a textual unit, the method comprising:
based on a plurality of word embedding vectors and a plurality of word scores, obtaining an embedding vector for the textual unit, wherein the embedding vector for the textual unit is based on a composite embedding vector and on information from a word score vector that includes the plurality of word scores, wherein the composite embedding vector is based on the plurality of word embedding vectors, and wherein obtaining the embedding vector for the textual unit comprises:
projecting the word score vector to obtain a projected vector having a dimensionality that is lower than a dimensionality of the word score vector; and
combining the composite embedding vector with the projected vector;
passing the embedding vector for the textual unit through at least one feed-forward layer to obtain a final layer output; and
performing a classification on the final layer output,
wherein, for each word among a plurality of words of the textual unit:
the plurality of word embedding vectors includes a corresponding word embedding vector, and
the plurality of word scores includes a corresponding word score.
2 . The method of claim 1 , wherein the at least one feed-forward layer comprises a multi-layer perceptron.
3 . The method of claim 1 , wherein performing the classification on the final layer output comprises applying a softmax function to the final layer output.
4 . The method of claim 1 , wherein the method comprises obtaining the plurality of word embedding vectors using a context-independent word embedding model.
5 . The method of claim 1 , wherein the method comprises obtaining the plurality of word embedding vectors using a trained FastText model.
6 . The method of claim 1 , wherein, for each word among the plurality of words, the corresponding word score is based on a term frequency of the word in the textual unit.
7 . The method of claim 1 , wherein, for each word among the plurality of words, the corresponding word score is based on a document frequency of the word.
8 . The method of claim 1 , wherein, for each word among the plurality of words, the corresponding word score is based on:
a term frequency of the word in the textual unit,
a document frequency of the word, and
one or more learned weighting parameters.
9 . The method of claim 1 , wherein, for each word among the plurality of words, the corresponding word score is based on a smooth inverse frequency of the word.
10 . The method of claim 1 , wherein obtaining the embedding vector for the textual unit comprises:
scaling each word embedding vector of the plurality of word embedding vectors with the corresponding word score of the plurality of word scores to obtain a plurality of scaled word embedding vectors, and
obtaining the embedding vector for the textual unit as a sum or average of the plurality of scaled word embedding vectors.
11 . The method of claim 1 , wherein the combining comprises at least one among concatenating, interpolating, gating, or attention.
12 . A system comprising:
one or more data processors; and
one or more non-transitory computer readable media storing instructions which, when executed by the one or more data processors, cause the one or more data processors to perform processing comprising:
based on a plurality of word embedding vectors and a plurality of word scores, obtaining an embedding vector for a textual unit, wherein the embedding vector for the textual unit is based on a composite embedding vector and on information from a word score vector that includes the plurality of word scores, wherein the composite embedding vector is based on the plurality of word embedding vectors, and wherein obtaining the embedding vector for the textual unit comprises:
projecting the word score vector to obtain a projected vector having a dimensionality that is lower than a dimensionality of the word score vector, and
combining the composite embedding vector with the projected vector;
passing the embedding vector for the textual unit through at least one feed-forward layer to obtain a final layer output; and
performing a classification on the final layer output,
wherein, for each word among a plurality of words of the textual unit:
the plurality of word embedding vectors includes a corresponding word embedding vector, and
the plurality of word scores includes a corresponding word score.
13 . The system of claim 12 , wherein the at least one feed-forward layer comprises a multi-layer perceptron.
14 . The system of claim 12 , wherein performing the classification on the final layer output comprises applying a softmax function to the final layer output.
15 . The system of claim 12 , wherein the processing comprises obtaining the plurality of word embedding vectors using a context-independent word embedding model.
16 . The system of claim 12 , wherein the processing comprises obtaining the plurality of word embedding vectors using a trained FastText model.
17 . The system of claim 12 , wherein, for each word among the plurality of words, the corresponding word score is based on a term frequency of the word in the textual unit.
18 . The system of claim 12 , wherein, for each word among the plurality of words, the corresponding word score is based on a document frequency of the word.
19 . A computer-program product tangibly embodied in one or more non-transitory machine-readable media, including instructions configured to cause one or more data processors to perform processing comprising:
based on a plurality of word embedding vectors and a plurality of word scores, obtaining an embedding vector for a textual unit, wherein the embedding vector for the textual unit is based on a composite embedding vector and on information from a word score vector that includes the plurality of word scores, wherein the composite embedding vector is based on a plurality of word embedding vectors, and wherein obtaining the embedding vector for the textual unit comprises:
projecting the word score vector to obtain a projected vector having a dimensionality that is lower than a dimensionality of the word score vector, and
combining the composite embedding vector with the projected vector;
passing the embedding vector for the textual unit through at least one feed-forward layer to obtain a final layer output; and
performing a classification on the final layer output,
wherein, for each word among a plurality of words of the textual unit:
the plurality of word embedding vectors includes a corresponding word embedding vector, and
the plurality of word scores includes a corresponding word score.
20 . The computer-program product of claim 19 , wherein the composite embedding vector is a sum or average of the plurality of word embedding vectors.