Generating vector representations of documents
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating document vector representations. One of the methods includes obtaining a new document; selecting a plurality of new document word sets; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system comprises: a document embedding layer and a classifier, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of new document word sets to the trained neural network system to determine the vector representation for the new document using gradient descent.
1. A method comprising:
obtaining a training document;
selecting a plurality of document vector training sets, each document vector training set comprising a predetermined number of surrounding words from the training document;
selecting a plurality of word vector training sets, each word vector training set comprising a respective document word from the training document and the predetermined number of surrounding words, wherein each surrounding word is a word that is within a window of the document word in the training document; and
training a neural network system on each of the plurality of document vector training sets and word vector training sets,
wherein the neural network system comprises:
a document embedding layer configured to receive an identifier for an input document and to map the identifier to a vector representation for the input document,
a word embedding layer configured to receive an input word and to map the input word to a vector representation of the input word, and
a classifier, wherein the classifier is configured to receive a vector representation and to generate a set of word scores from the received vector representation,
wherein the set of word scores includes a respective word score for each word in a pre-determined set of words,
wherein each of the respective word scores represents a predicted likelihood that a corresponding word is a word in the input document, and
wherein training the neural network system on each of the plurality of document vector training sets and word vector training sets comprises jointly learning parameters of the document embedding layer, parameters of the word embedding layer and parameters of the classifier by performing a respective iteration of gradient descent for each document vector training set, and performing a respective iteration of gradient descent for each word vector training set,
wherein performing the respective iteration of gradient descent for each document vector training set comprises:
processing an identifier for the training document using the document embedding layer to map the identifier to a vector representation for the training document;
processing the vector representation for the training document using the classifier to generate a set of word scores; and
determining, for each of the words from the document vector training set, a respective error between the set of word scores generated by the classifier and a target output for the word from the document vector training set that indicates that the word appears in the training document;
wherein performing the respective iteration of gradient descent for each word vector training set comprises:
processing the document word in the word vector training set using the word embedding layer to map the document word to a vector representation for the document word;
processing the vector representation for the document word using the classifier to generate a set of word scores; and
determining, for each surrounding word from the word vector training set, a respective error between the set of word scores generated by the classifier and a target output for each surrounding word from the word vector training set;
and wherein jointly learning parameters of the document embedding layer, parameters of the word embedding layer and parameters of the classifier comprises:
adjusting current values of the parameters of the classifier using the respective errors for each document vector training set and the respective errors for each word vector training set;
adjusting current values of the parameters of the word embedding layer using the respective errors for each word vector training set using backpropagation; and
adjusting current values of the parameters of the document embedding layer using the respective errors for each document vector training set by backpropagation.
2. The method of claim 1 , further comprising:
providing a new document to the trained neural network system; and
determining a document vector representation for the new document using the trained neural network system.
3. The method of claim 2 , wherein determining a document vector representation for the new document comprises:
selecting multiple sets of words from the new document, wherein each set of words comprises same fixed number of words; and
processing each set of words using the trained neural network system to iteratively determine the document vector representation for the new document.
4. The method of claim 2 , further comprising:
associating the document vector representation of the new document with the new document stored in a repository.
5. The method of claim 2 , further comprising:
providing the document vector representation for the new document to another machine learning system that has been configured to classify input documents.
6. The method of claim 5 , wherein the other machine learning system comprises at least one of a logistic regression system, a Support Vector Machine (SVM) system, and a k-means system.
7. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
obtaining a training document;
selecting a plurality of document vector training sets, each document vector training set comprising a predetermined number of surrounding words from the training document;
selecting a plurality of word vector training sets, each word vector training set comprising a respective document word from the training document and the predetermined number of surrounding words, wherein each surrounding word is a word that is within a window of the document word in the training document; and
training a neural network system on each of the plurality of document vector training sets and word vector training sets,
wherein the neural network system comprises:
a document embedding layer configured to receive an identifier for an input document and to map the identifier to a vector representation for the input document,
a word embedding layer configured to receive an input word and to map the input word to a vector representation of the input word, and
a classifier, wherein the classifier is configured to receive a vector representation and to generate a set of word scores from the received vector representation,
wherein the set of word scores includes a respective word score for each word in a pre-determined set of words,
wherein each of the respective word scores represents a predicted likelihood that a corresponding word is a word in the input document, and
wherein training the neural network system on each of the plurality of document vector training sets and word vector training sets comprises jointly learning parameters of the document embedding layer, parameters of the word embedding layer and parameters of the classifier by performing a respective iteration of gradient descent for each document vector training set, and performing a respective iteration of gradient descent for each word vector training set,
wherein performing the respective iteration of gradient descent for each document vector training set comprises:
processing an identifier for the training document using the document embedding layer to map the identifier to a vector representation for the training document;
processing the vector representation for the training document using the classifier to generate a set of word scores; and
determining, for each of the words from the document vector training set, a respective error between the set of word scores generated by the classifier and a target output for the word from the document vector training set that indicates that the word appears in the training document;
wherein performing the respective iteration of gradient descent for each word vector training set comprises:
processing the document word in the word vector training set using the word embedding layer to map the document word to a vector representation for the document word;
processing the vector representation for the document word using the classifier to generate a set of word scores; and
determining, for each surrounding word from the word vector training set, a respective error between the set of word scores generated by the classifier and a target output for each surrounding word from the word vector training set;
and wherein jointly learning parameters of the document embedding layer, parameters of the word embedding layer and parameters of the classifier comprises:
adjusting current values of the parameters of the classifier using the respective errors for each document vector training set and the respective errors for each word vector training set;
adjusting current values of the parameters of the word embedding layer using the respective errors for each word vector training set using backpropagation; and
adjusting current values of the parameters of the document embedding layer using the respective errors for each document vector training set by backpropagation.
8. The system of claim 7 , the operations further comprise:
providing a new document to the trained neural network system; and
determining a document vector representation for the new document using the trained neural network system.
9. The system of claim 8 , wherein determining a document vector representation for the new document comprises:
selecting multiple sets of words from the new document, wherein each set of words comprises same fixed number of words; and
processing each set of words using the trained neural network system to iteratively determine the document vector representation for the new document.
10. The system of claim 8 , the operations further comprise:
associating the document vector representation of the new document with the new document stored in a repository.
11. The system of claim 8 , the operations further comprise:
providing the document vector representation for the new document to another machine learning system that has been configured to classify input documents.
12. The system of claim 11 , wherein the other machine learning system comprises at least one of a logistic regression system, a Support Vector Machine (SVM) system, and a k-means system.
13. A computer program product encoded on one or more non-transitory computer readable media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining a training document;
selecting a plurality of document vector training sets, each document vector training set comprising a predetermined number of surrounding words from the training document;
selecting a plurality of word vector training sets, each word vector training set comprising a respective document word from the training document and the predetermined number of surrounding words, wherein each surrounding word is a word that is within a window of the document word in the training document; and
training a neural network system on each of the plurality of document vector training sets and word vector training sets,
wherein the neural network system comprises:
a document embedding layer configured to receive an identifier for an input document and to map the identifier to a vector representation for the input document,
a word embedding layer configured to receive an input word and to map the input word to a vector representation of the input word, and
a classifier, wherein the classifier is configured to receive a vector representation and to generate a set of word scores from the received vector representation,
wherein the set of word scores includes a respective word score for each word in a pre-determined set of words,
wherein each of the respective word scores represents a predicted likelihood that a corresponding word is a word in the input document, and
wherein training the neural network system on each of the plurality of document vector training sets and word vector training sets comprises jointly learning parameters of the document embedding layer, parameters of the word embedding layer and parameters of the classifier by performing a respective iteration of gradient descent for each document vector training set, and performing a respective iteration of gradient descent for each word vector training set,
wherein performing the respective iteration of gradient descent for each document vector training set comprises:
processing an identifier for the training document using the document embedding layer to map the identifier to a vector representation for the training document;
processing the vector representation for the training document using the classifier to generate a set of word scores; and
determining, for each of the words from the document vector training set, a respective error between the set of word scores generated by the classifier and a target output for the word from the document vector training set that indicates that the word appears in the training document;
wherein performing the respective iteration of gradient descent for each word vector training set comprises:
processing the document word in the word vector training set using the word embedding layer to map the document word to a vector representation for the document word;
processing the vector representation for the document word using the classifier to generate a set of word scores; and
determining, for each surrounding word from the word vector training set, a respective error between the set of word scores generated by the classifier and a target output for each surrounding word from the word vector training set;
and wherein jointly learning parameters of the document embedding layer, parameters of the word embedding layer and parameters of the classifier comprises:
adjusting current values of the parameters of the classifier using the respective errors for each document vector training set and the respective errors for each word vector training set;
adjusting current values of the parameters of the word embedding layer using the respective errors for each word vector training set using backpropagation; and
adjusting current values of the parameters of the document embedding layer using the respective errors for each document vector training set by backpropagation.
14. The computer program product of claim 13 , the operations further comprise:
providing a new document to the trained neural network system; and
determining a document vector representation for the new document using the trained neural network system.
15. The computer program product of claim 14 , wherein determining a document vector representation for the new document comprises:
selecting multiple sets of words from the new document, wherein each set of words comprises same fixed number of words; and
processing each set of words using the trained neural network system to iteratively determine the document vector representation for the new document.
16. The computer program product of claim 14 , the operations further comprise:
associating the document vector representation of the new document with the new document stored in a repository.
17. The computer program product of claim 14 , the operations further comprise:
providing the document vector representation for the new document to another machine learning system that has been configured to classify input documents.
18. The computer program product of claim 17 , wherein the other machine learning system comprises at least one of a logistic regression system, a Support Vector Machine (SVM) system, and a k-means system.