Document classification with weighted supervised n-gram embedding
View Patent ↗Methods and systems for document classification include embedding n-grams from an input text in a latent space, embedding the input text in the latent space based on the embedded n-grams and weighting said n-grams according to spatial evidence of the respective n-grams in the input text, classifying the document along one or more axes, and adjusting weights used to weight the n-grams based on the output of the classifying step.
1. A method for document classification, comprising:
embedding n-grams from an input text in a latent space;
embedding the input text in the latent space based on the embedded n-grams and weighting the n-grams according to a non-linear function
q
j
=
1
Q
∑
k
=
1
K
sigmoid
(
a
k
·
j
N
+
b
k
)
,
using a mixture model on a relative position of the n-grams in the input text, where a k and b k are parameters to be learned,
Q
=
∑
j
=
1
N
q
j
and K specify a number of mixture quantities, sigmoid (•) is a non-linear transfer function, q j is the weight associated with a j th n-gram, j signifies the position of an n-gram in the input text, and N is the position of a final n-gram in the input text;
classifying the document along one or more axes using a processor; and
adjusting weights used to weight the n-grams based on the output of the classifying step.
2. The method of claim 1 , wherein embedding the input text in the latent space is calculated as a weighted sum over the embedded n-grams in the input text.
3. The method of claim 1 , wherein a weight of each n-gram is modeled as a function of a relative position of each n-gram in the input text and an embedding representation of each n-gram.
4. The method of claim 1 , wherein embedding the input text in the latent space further comprises:
dividing the input text into sub-parts;
forming an embedded representation of each of the sub-parts based on embedded n-grams in each respective sub-part; and
concatenating the sub-parts to form an embedded representation of the full input text.
5. The method of claim 4 , wherein forming the embedded representation of each subpart includes calculating a weighted sum over the n-grams in the subpart.
6. The method of claim 1 , wherein embedding the input text in the latent space comprises calculating a weighted sum over the n-grams in the input text.
7. The method of claim 1 , wherein weights for each n-gram are learned by optimizing over a set of training documents with known class labels.
8. The method of claim 7 , wherein the weights are learned using a stochastic gradient descent.
9. The method of claim 1 , wherein classifying includes applying a classification having three or more classes.
10. A system for document classification, comprising:
an n-gram embedding module configured to embed n-grams from an input text in a latent space;
a document embedding module configured to embed the input the input text in the latent space based on the embedded n-grams, weighted according to a non-linear function
q
j
=
1
Q
∑
k
=
1
K
sigmoid
(
a
k
·
j
N
+
b
k
)
,
using a mixture model on a relative position of the n-grams in the input text, where a k and b k are parameters to be learned,
Q
=
∑
j
=
1
N
q
j
and K specify a number of mixture quantities, sigmoid (•) is a non-linear transfer function, q j is the weight associated with a j th n-gram, j signifies the position of an n-ram in the input text and N is the position of a final n-gram in the input text;
a classifier configured to classify the document along one or more axes using a processor; and
a weight learning module configured to adjust the weights for the n-grams based on the output of the classifying step.
11. The system of claim 10 , wherein the document embedding module is further configured to embed the input text in the latent space as a weighted sum over the embedded n-grams in the input text.
12. The system of claim 10 , wherein the weight of an n-gram is modeled as a function of both a relative position of the n-gram in the document and an embedding representation of the n-gram.
13. The system of claim 10 , wherein the document embedding module is further configured to:
divide the input text into sub-parts;
form an embedded representation of each of the sub-parts based on embedded n-grams in each respective sub-part; and
concatenate the sub-parts to form an embedded representation of the full input text.
14. The system of claim 13 , wherein the document embedding module is further configured to form the embedded representation of each subpart by calculating a weighted sum over the n-grams in the subpart.
15. The system of claim 10 , wherein the document embedding module is further configured to form the embedded representation of each subpart by calculating a weighted sum over the n-grams in the subpart.
16. The system of claim 10 , wherein the weight learning module is further configured to learn weights for each n-gram by optimizing over a set of training documents with known class labels.
17. The system of claim 16 , wherein the weight learning module is configured to learn the weights using a stochastic gradient descent.
18. The system of claim 10 , wherein the classifier is configured to apply a classification having three or more classes.