Generating vector representations of documents
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating document vector representations. One of the methods includes obtaining a new document; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system has been trained to receive an input document and a sequence of words from the input document and to generate a respective word score for each word in a set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence in the input document, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of sequences of words to the trained neural network system to determine the vector representation for the new document using gradient descent.
1. A method comprising:
obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that is in another sequence of words in the new document and that follows a last word in the sequence of words in the new document;
generating a vector representation of the new document using a trained neural network system, wherein generating the vector representation of the new document using the trained neural network system comprises, for each iteration step of multiple iteration steps:
obtaining a current sequence of words from the plurality of sequences of words;
processing (i) data identifying the new document and (ii) the current sequence of words by the trained neural network system having an embedding layer and one or more other layers and in accordance with (i) trained values of a set of word parameters of the embedding layer and (ii) current values of a set of document parameters of the embedding layer to generate a respective word score for each word in a pre-determined set of words;
computing a gradient with respect to the vector representation of an error function that measures an error between the respective word scores and a target set of word scores that identifies a word that is in another sequence of words in the new document and that follows a last word in the current sequence of words in the new document; and
training the trained neural network system on the new document to adjust the current values of the set of document parameters of the embedding layer of the trained neural network system based on the gradient using gradient descent while holding the trained values of the set of word parameters of the embedding layer of the trained neural network system fixed; and
processing, by a text classification system, an input comprising the vector representation of the new document to generate a classification output for the new document.
2. The method of claim 1 , wherein the classification output comprises a respective score for each of a set of topics, with each score representing an estimated likelihood that the new document is about or relates to the corresponding topic.
3. The method of claim 1 , wherein the text classification system comprises a machine learning system.
4. The method of claim 3 , wherein the machine learning system comprises one of: a logistic regression system, a Support Vector Machines (SVM) system, or a k-means system.
5. The method of claim 1 , further comprising associating the vector representation with the document in a repository.
6. The method of claim 1 , wherein the trained values of the set of word parameters are determined from a training of the trained neural network system on different training documents.
7. The method of claim 6 , wherein the different training documents are unlabeled.
8. The method of claim 1 , wherein obtaining the current sequences of words from the plurality of sequences of words comprises:
applying a sliding window to the new document to extract each possible sequence of a predetermined fixed length from the new document.
9. The method of claim 1 , wherein the document comprises a web page.
10. A system comprising:
one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that is in another sequence of words in the new document and that follows a last word in the sequence of words in the new document;
generating a vector representation of the new document using a trained neural network system, wherein generating the vector representation of the new document using the trained neural network system comprises, for each iteration step of multiple iteration steps:
obtaining a current sequence of words from the plurality of sequences of words;
processing (i) data identifying the new document and (ii) the current sequence of words by the trained neural network system having an embedding layer and one or more other layers and in accordance with (i) trained values of a set of word parameters of the embedding layer and (ii) current values of a set of document parameters of the embedding layer to generate a respective word score for each word in a pre-determined set of words;
computing a gradient with respect to the vector representation of an error function that measures an error between the respective word scores and a target set of word scores that identifies a word that is in another sequence of words in the new document and that follows a last word in the current sequence of words in the new document; and
training the trained neural network system on the new document to adjust the current values of the set of document parameters of the embedding layer of the trained neural network system based on the gradient using gradient descent while holding the trained values of the set of word parameters of the embedding layer of the trained neural network system fixed; and
processing, by a text classification system, an input comprising the vector representation of the new document to generate a classification output for the new document.
11. The system of claim 10 , wherein the classification output comprises a respective score for each of a set of topics, with each score representing an estimated likelihood that the new document is about or relates to the corresponding topic.
12. The system of claim 10 , wherein the text classification system comprises a machine learning system.
13. The system of claim 12 , wherein the machine learning system comprises one of: a logistic regression system, a Support Vector Machines (SVM) system, or a k-means system.
14. The system of claim 10 , wherein the operations further comprise associating the vector representation with the document in a repository.
15. The system of claim 10 , wherein the trained values of the set of word parameters are determined from a training of the trained neural network system on different training documents.
16. The system of claim 15 , wherein the different training documents are unlabeled.
17. The system of claim 10 , wherein obtaining the current sequences of words from the plurality of sequences of words comprises:
applying a sliding window to the new document to extract each possible sequence of a predetermined fixed length from the new document.
18. The system of claim 10 , wherein the document comprises a web page.
19. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that is in another sequence of words in the new document and that follows a last word in the sequence of words in the new document;
generating a vector representation of the new document using a trained neural network system, wherein generating the vector representation of the new document using the trained neural network system comprises, for each iteration step of multiple iteration steps:
obtaining a current sequence of words from the plurality of sequences of words;
processing (i) data identifying the new document and (ii) the current sequence of words by the trained neural network system having an embedding layer and one or more other layers and in accordance with (i) trained values of a set of word parameters of the embedding layer and (ii) current values of a set of document parameters of the embedding layer to generate a respective word score for each word in a pre-determined set of words;
computing a gradient with respect to the vector representation of an error function that measures an error between the respective word scores and a target set of word scores that identifies a word that is in another sequence of words in the new document and that follows a last word in the current sequence of words in the new document; and
training the trained neural network system on the new document to adjust the current values of the set of document parameters of the embedding layer of the trained neural network system based on the gradient using gradient descent while holding the trained values of the set of word parameters of the embedding layer of the trained neural network system fixed; and
processing, by a text classification system, an input comprising the vector representation of the new document to generate a classification output for the new document.
20. The storage media of claim 19 , wherein the classification output comprises a respective score for each of a set of topics, with each score representing an estimated likelihood that the new document is about or relates to the corresponding topic.
21. The method of claim 1 , wherein training the trained neural network system on the new document comprises holding trained values of parameters of the one or more other layers of the trained neural network system fixed.