Document information extraction without additional annotations
Disclosed herein are system, method, and computer program product embodiments for document information extraction without additional annotations. An embodiment operates by receiving an input representing a document and a key. The embodiment processes the input using a convolutional neural network to obtain a feature map. The embodiment combines the feature map with positional information to obtain a spatial-aware feature map. The embodiment then repeatedly performs the following decoding process: generate attention weights, generate a context vector based on the spatial-aware feature map and the generated attention weights using an attention layer, process the context vector, the key, and an input vector using a recurrent neural network (RNN) to obtain a RNN state, and generate an output vector based on the RNN state and the context vector using a projection layer. The embodiment then extracts a field based on the result of the decoding process.
1. A computer implemented method for extracting a field from a document, comprising:
receiving, by at least one processor, the document and a key, wherein the key specifies the field to extract from the document, the key is in textual format, the field is a plurality of characters, and the document is an image representing a plurality of fields;
processing, by a convolutional neural network (CNN) of an encoder, the document, thereby obtaining a feature map;
combining, by the encoder, the feature map with positional information for each feature in the feature map, thereby obtaining a spatial-aware feature map;
processing, using a recurrent neural network (RNN) of a decoder, the spatial-aware feature map and the key, thereby extracting the field in the document corresponding to the key, wherein the processing the spatial-aware feature map and the key comprises:
initializing, a first RNN state associated with the RNN of the decoder, a first set of attention weights for an attention layer, and a first output vector;
generating a second set of attention weights for the attention layer based on the spatial-aware feature map, the key, the first RNN state associated with the RNN, the first set of attention weights for the attention layer, and the first output vector;
generating a context vector based on the spatial-aware feature map and the second set of attention weights using the attention layer;
processing the context vector, the key, and an input vector using the RNN to obtain a second RNN state associated with the RNN;
generating a second output vector based on the second RNN state and the context vector using a projection layer;
storing the second output vector in a list of output vectors;
repeating, until the second output vector corresponds to an end token, the generating the second set of attention weights, the generating the context vector, the processing the context vector, the generating the second output vector, and the storing the second output vector with the second set of attention weights set to a value of the first set of attention weights, the second RNN state set to a value of the first RNN state, the second output vector set to a value of the first output vector, and the second output vector set to a value of the input vector; and
extracting only the field from the document based on the list of output vectors, wherein each output vector of the list of output vectors is derived from the key.
2. The method of claim 1 , further comprising:
generating the positional information for each feature in the feature map as a set of one-hot vectors.
3. The method of claim 1 , further comprising:
generating the positional information for each feature in the feature map as a set of scalar values.
4. The method of claim 1 , further comprising:
generating the positional information for each feature in the feature map as a fixed length vector, wherein the fixed length vector is generated by training the attention layer, the RNN, and the projection layer.
5. The method of claim 1 , wherein the attention layer is a sum-attention layer, and the RNN is a long short-term memory (LSTM).
6. The method of claim 1 , further comprising:
encoding, by the at least one processor, the key using a one-hot vector or a fixed length vector, wherein the fixed length vector is generated by training the attention layer, the RNN, and the projection layer.
7. The method of claim 1 , further comprising:
initializing a set of weights of the CNN with a set of weights of a pre-trained optical character recognition (OCR) model;
initializing the RNN and the projection layer with values sampled from a random distribution;
training the attention layer, the RNN, and the projection layer based on the set of weights of the CNN being fixed;
detecting an increase in performance of the attention layer, the RNN, and the projection layer above a threshold value; and
in response to the detecting, training the CNN, the attention layer, the RNN, and the projection layer until a loss on a validation set stops decreasing.
8. A system, comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive a document and a key, wherein the key specifies a field to extract from the document, the key is in textual format, the field is a plurality of characters, and the document is an image representing a plurality of fields;
process, by a convolutional neural network (CNN) of an encoder, the document, thereby obtaining a feature map;
combine, by the encoder, the feature map with positional information for each feature in the feature map, thereby obtaining a spatial-aware feature map;
process, using a recurrent neural network (RNN) of a decoder, the spatial-aware feature map and the key, thereby extracting the field in the document corresponding to the key, wherein the processing the spatial-aware feature map and the key comprises:
initializing a first RNN state associated with the RNN of the decoder, a first set of attention weights for an attention layer, and a first output vector;
generating a second set of attention weights for the attention layer based on the spatial-aware feature map, the key, the first RNN state associated with the RNN, the first set of attention weights for the attention layer, and the first output vector;
generating a context vector based on the spatial-aware feature map and the second set of attention weights using the attention layer;
processing the context vector, the key, and an input vector using the RNN to obtain a second RNN state associated with the RNN;
generating a second output vector based on the second RNN state and the context vector using a projection layer;
storing the second output vector in a list of output vectors;
repeating, until the second output vector corresponds to an end token, the generate the second set of attention weights, the generate the context vector, the process the context vector, the generate the second output vector, and the store the second output vector with the second set of attention weights set to a value of the first set of attention weights, the second RNN state set to a value of the first RNN state, the second output vector set to a value of the first output vector, and the second output vector set to a value of the input; and
extracting only the field from the document based on the list of output vectors, wherein each output vector of the list of output vectors is derived from the key.
9. The system of claim 8 , the at least one processor further configured to:
generate the positional information for each feature in the feature map as a set of one-hot vectors.
10. The system of claim 8 , the at least one processor further configured to:
generate the positional information for each feature in the feature map as a set of scalar values.
11. The system of claim 8 , the at least one processor further configured to:
generate the positional information for each feature in the feature map as a fixed length vector, wherein the fixed length vector is generated by training the attention layer, the RNN, and the projection layer.
12. The system of claim 8 , wherein the attention layer is a sum-attention layer, and the RNN is a long short-term memory (LSTM).
13. The system of claim 8 , the at least one processor further configured to:
encode the key using a one-hot vector or a fixed length vector, wherein the fixed length vector is generated by training the attention layer, the RNN, and the projection layer.
14. The system of claim 8 , the at least one processor further configured to:
initialize a set of weights of the CNN with a set of weights of a pre-trained optical character recognition (OCR) model;
initialize the RNN and the projection layer with values sampled from a random distribution;
train the attention layer, the RNN, and the projection layer based on the set of weights of the CNN being fixed;
detect an increase in performance of the attention layer, the RNN, and the projection layer above a threshold value; and
in response to detection of the increase in performance of the attention layer, train the CNN, the attention layer, the RNN, and the projection layer until a loss on a validation set stops decreasing.
15. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
receiving a document and a key, wherein the key specifies a field to extract from the document, the key is in textual format, the field is a plurality of characters, and the document is an image representing a plurality of fields;
processing, by a convolutional neural network (CNN) of an encoder, the document, thereby obtaining a feature map;
combining, by the encoder, the feature map with positional information for each feature in the feature map, thereby obtaining a spatial-aware feature map;
processing, using a recurrent neural network (RNN) of a decoder, the spatial-aware feature map and the key, thereby extracting the field in the document corresponding to the key, wherein the processing the spatial-aware feature map and the key comprises:
initializing a first RNN state associated with the RNN of the decoder, a first set of attention weights for an attention layer, and a first output vector;
generating a second set of attention weights for the attention layer based on the spatial-aware feature map, the key, the first RNN state associated with the RNN, the first set of attention weights for the attention layer, and the first output vector;
generating a context vector based on the spatial-aware feature map and the second set of attention weights using the attention layer;
processing the context vector, the key, and an input vector using the RNN to obtain a second RNN state associated with the RNN;
generating a second output vector based on the second RNN state and the context vector using a projection layer;
storing the second output vector in a list of output vectors;
repeating, until the second output vector corresponds to an end token, the generating the second set of attention weights, the generating the context vector, the processing the context vector, the generating the second output vector, and the storing the second output vector with the second set of attention weights set to a value of the first set of attention weights, the second RNN state set to a value of the first RNN state, the second output vector set to a value of the first output vector, and the second output vector set to a value of the input vector; and
extracting only the field from the document based on the list of output vectors, wherein each output vector of the list of output vectors is derived from the key.
16. The non-transitory computer-readable device of claim 15 , the operations further comprising:
generating the positional information for each feature in the feature map as a set of one-hot vectors.
17. The non-transitory computer-readable device of claim 15 , the operations further comprising:
generating the positional information for each feature in the feature map as a set of scalar values.
18. The non-transitory computer-readable device of claim 15 , the operations further comprising:
generating the positional information for each feature in the feature map as a fixed length vector, wherein the fixed length vector is generated by training the attention layer, the RNN, and the projection layer.
19. The non-transitory computer-readable device of claim 15 , wherein the attention layer is a sum-attention layer, and the RNN is a long short-term memory (LSTM).
20. The non-transitory computer-readable device of claim 15 , the operations further comprising:
initializing a set of weights of the CNN with a set of weights of a pre-trained optical character recognition (OCR) model;
initializing the RNN and the projection layer with values sampled from a random distribution;
training the attention layer, the RNN, and the projection layer based on the set of weights of the CNN being fixed;
detecting an increase in performance of the attention layer, the RNN, and the projection layer above a threshold value; and
in response to the detecting, training the CNN, the attention layer, the RNN, and the projection layer until a loss on a validation set stops decreasing.