Learning document embeddings with convolutional neural network architectures
A document analysis system trains a document embedding model configured to receive a set of word embeddings for an ordered set of words in a document and generate a document embedding for the document. The document embedding is a representation of the document in a latent space that characterizes the document with respect to properties such as structure, content, and sentiment. The document embedding may represent a prediction of a set of words that follow the last word in the ordered set of words of the document. The document embedding model may be associated with a convolutional neural network (CNN) architecture that includes one or more convolutional layers. The CNN architecture of the document embedding model allows the document analysis system to overcome various difficulties of existing document embedding models, and allows the document analysis system to easily process variable-length documents that include a variable number of words.
1. A method of training a document embedding model for generating a document embedding for a corresponding document, the method comprising:
identifying a batch of training documents, each training document including an ordered set of words;
for each training document in the batch of training documents:
determining, for each of the ordered set of words included in the training document, a corresponding word embedding for a word that represents the word in a latent space;
identifying a first subset of words in the training document and a second subset of words in the training document subsequent to the first subset of words;
generating an estimated document embedding for the training document by applying the document embedding model with an estimated first set of parameters to the word embeddings for the first subset of words; and
determining a loss function indicating distances between the estimated document embedding and the word embeddings for the second subset of words; and
updating the first set of parameters for the document embedding model to reduce the loss function relative to the estimated first set of parameters of the document embedding model.
2. The method of claim 1 , determining the loss function further comprises:
identifying a negative set of words that are not included in the training document; and
determining, for each of the negative set of words, a corresponding word embedding for the word, wherein the loss function further indicates distances between the estimated embedding and the word embeddings for the negative set of words.
3. The method of claim 1 , wherein determining the word embeddings for the ordered set of words comprises:
applying, for each of the ordered set of words, a word embedding model with an estimated second set of parameters to the word, and wherein the method further comprises updating the second set of parameters for the word embedding model to reduce the loss function relative to the estimated second set of parameters of the word embedding model.
4. The method of claim 1 , wherein a dimensionality of the latent space of the word embedding is same as a dimensionality of a latent space of the estimated document embedding.
5. The method of claim 1 , wherein the document embedding model has a convolutional neural network architecture, and wherein generating the estimated document embedding further comprises generating a fixed vector with a fixed-number of elements at an aggregation layer placed after a series of convolutional layers, wherein the fixed vector is generated by performing an aggregation operation to an input at a layer before the aggregation layer.
6. The method of claim 5 , wherein the layer before the aggregation layer is a convolutional layer associated with a feature map, and wherein the fixed vector is generated by performing the aggregation operation to elements of the feature map associated with each particular kernel of the convolutional layer.
7. The method of claim 5 , wherein generating the document embedding further comprises generating outputs at one or more fully connected layers of the convolutional neural network architecture that are placed after the aggregation layer, the outputs generated by:
for each fully connected layer, generating an output at the fully connected layer by applying a set of parameters to each input at another layer before the fully connected layer.
8. A non-transitory computer-readable medium containing instructions for execution on a processor, the instructions comprising:
identifying a batch of training documents, each training document including an ordered set of words;
for each training document in the batch of training documents:
determining, for each of the ordered set of words included in the training document, a corresponding word embedding for a word that represents the word in a latent space;
identifying a first subset of words in the training document and a second subset of words in the training document subsequent to the first subset of words;
generating an estimated document embedding for the training document by applying a document embedding model with an estimated first set of parameters to the word embeddings for the first subset of words; and
determining a loss function indicating distances between the estimated document embedding and the word embeddings for the second subset of words; and
updating the first set of parameters for the document embedding model to reduce the loss function relative to the estimated first set of parameters of the document embedding model.
9. The computer-readable medium of claim 8 , wherein determining the loss function further comprises:
identifying a negative set of words that are not included in the training document; and
determining, for each of the negative set of words, a corresponding word embedding for the word, wherein the loss function further indicates distances between the estimated embedding and the word embeddings for the negative set of words.
10. The computer-readable medium of claim 8 , wherein determining the word embeddings for the ordered set of words comprises:
applying, for each of the ordered set of words, a word embedding model with an estimated second set of parameters to the word, and wherein the instructions further comprise updating the second set of parameters for the word embedding model to reduce the loss function relative to the estimated second set of parameters of the word embedding model.
11. The computer-readable medium of claim 8 , wherein a dimensionality of the latent space of the word embedding is same as a dimensionality of a latent space of the estimated document embedding.
12. The computer-readable medium of claim 8 , wherein the document embedding model has a convolutional neural network architecture, and wherein generating the estimated document embedding further comprises generating a fixed vector with a fixed-number of elements at an aggregation layer placed after a series of convolutional layers, wherein the fixed vector is generated by performing an aggregation operation to an input at a layer before the aggregation layer.
13. The computer-readable medium of claim 12 , wherein generating the document embedding further comprises generating outputs at one or more fully connected layers of the convolutional neural network architecture that are placed after the aggregation layer, the outputs generated by:
for each fully connected layer, generating an output at the fully connected layer by applying a set of parameters to each input at another layer before the fully connected layer.
14. A machine-learned document embedding model stored on a non-transitory computer-readable medium, wherein the document embedding model is manufactured by a process comprising:
identifying a batch of training documents, each training document including an ordered set of words;
for each training document in the batch of training documents:
determining, for each of the ordered set of words included in the training document, a corresponding word embedding for a word that represents the word in a latent space;
identifying a first subset of words in the training document and a second subset of words in the training document subsequent to the first subset of words;
generating an estimated document embedding for the training document by applying the document embedding model with an estimated first set of parameters to the word embeddings for the first subset of words; and
determining a loss function indicating distances between the estimated document embedding and the word embeddings for the second subset of words;
updating the first set of parameters for the document embedding model to reduce the loss function relative to the estimated first set of parameters of the document embedding model; and
storing the set of parameters on the computer-readable medium as parameters of the document embedding model.
15. The machine-learned document embedding model of claim 14 , wherein determining the loss function further comprises:
identifying a negative set of words that are not included in the training document; and
determining, for each of the negative set of words, a corresponding word embedding for the word, wherein the loss function further indicates distances between the estimated embedding and the word embeddings for the negative set of words.
16. The machine-learned document embedding model of claim 14 , wherein determining the word embeddings for the ordered set of words comprises:
applying, for each of the ordered set of words, a word embedding model with an estimated second set of parameters to the word, and wherein the process further comprises updating the second set of parameters for the word embedding model to reduce the loss function relative to the estimated second set of parameters of the word embedding model.
17. The machine-learned document embedding model of claim 14 , wherein a dimensionality of the latent space of the word embedding is same as a dimensionality of a latent space of the estimated document embedding.
18. The machine-learned document embedding model of claim 14 , wherein the document embedding model has a convolutional neural network architecture, and wherein generating the estimated document embedding further comprises generating a fixed vector with a fixed-number of elements at an aggregation layer placed after a series of convolutional layers, wherein the fixed vector is generated by performing an aggregation operation to an input at a layer before the aggregation layer.
19. The machine-learned document embedding model of claim 18 , wherein the layer before the aggregation layer is a convolutional layer associated with a feature map, and wherein the fixed vector is generated by performing the aggregation operation to elements of the feature map associated with each particular kernel of the convolutional layer.
20. The machine-learned document embedding model of claim 18 , wherein generating the document embedding further comprises generating outputs at one or more fully connected layers of the convolutional neural network architecture that are placed after the aggregation layer, the outputs generated by:
for each fully connected layer, generating an output at the fully connected layer by applying a set of parameters to each input at another layer before the fully connected layer.