End to end trainable document extraction
A processor may receive an image and identify a plurality of characters in the image using a machine learning (ML) model. The processor may generate at least one word-level bounding box indicating one or more words including at least a subset of the plurality of characters and/or may generate at least one field-level bounding box indicating at least one field including at least a subset of the one or more words. The processor may overlay the at least one word-level bounding box and the at least one field-level bounding box on the image to form a masked image including a plurality of optically-recognized characters and one or more predicted fields for at least a subset of the plurality of optically-recognized characters.
1. A method comprising:
receiving, by a processor, an image;
generating, by the processor, at least one field-level bounding box indicating at least one field including at least a subset of one or more characters in the image, the generating of the at least one field-level bounding box comprising:
predicting, for one or more pixels, a field-level bounding box or a background region to which the pixel belongs, and
for each pixel belonging to a respective field-level bounding box, determining an offset of the pixel to a center of the respective field-level bounding box; and
overlaying, by the processor, the at least one field-level bounding box on the image to form a masked image including a plurality of characters and one or more predicted fields for at least a subset of the plurality of characters.
2. The method of claim 1 , further comprising:
extracting, by the processor, information contained within the one or more predicted fields of the masked image; and
populating, by the processor, one or more fillable fields corresponding to the one or more predicted fields with the information.
3. The method of claim 1 , wherein the predicting comprises:
applying a directed graph algorithm to identify a direction from the pixel towards the center; and
applying a non-maximum suppression algorithm to remove at least one redundancy.
4. The method of claim 1 , wherein the generating of the at least one field-level bounding box comprises, for each respective field-level bounding box, setting a height and width of the at least one field-level bounding box to cause the at least one field-level bounding box to surround each pixel predicted to belong to the respective field-level bounding box.
5. The method of claim 1 , further comprising identifying, by the processor, the one or more characters in the image using a machine learning (ML) model.
6. The method of claim 1 , wherein the overlaying comprises:
forming the at least one field-level bounding box into a bounding box mask, wherein the at least one field-level bounding box is located in one or more positions corresponding to the groups of characters in the image; and
overlaying the bounding box mask on the image.
7. The method of claim 1 , wherein the ML model comprises at least one mapping function used by the processor to perform the identifying and configured from training data supplied to the ML model.
8. A method comprising:
training, by a processor, a machine learning (ML) model using a labeled training data set;
receiving, by the processor, an image;
generating, by the processor, at least one field-level bounding box indicating at least one field including at least a subset of one or more characters in the image, the generating of the at least one field-level bounding box comprising:
predicting, for each pixel, a field-level bounding box or a background region to which the pixel belongs, and
for each pixel belonging to a respective field-level bounding box, determining an offset of the pixel to a center of the respective field-level bounding box; and
overlaying, by the processor, the at least one field-level bounding box on the image to form a masked image including a plurality of characters and one or more predicted fields for at least a subset of the plurality of characters.
9. The method of claim 8 , wherein the labeled training data set includes pixel-level labels of at least one of character identity, bounding box size, center offset to character, center offset to word, and center offset to field.
10. The method of claim 8 , wherein the training comprises a deep neural network training process including a cross-entropy loss computation for categorical predictions and a regression loss computation for numerical predictions.
11. The method of claim 8 , further comprising:
extracting, by the processor, information contained within the one or more predicted fields of the masked image; and
populating, by the processor, one or more fillable fields corresponding to the one or more predicted fields with the information.
12. The method of claim 8 , wherein the predicting comprises:
applying a directed graph algorithm to identify a direction from the pixel towards the center; and
applying a non-maximum suppression algorithm to remove at least one redundancy.
13. The method of claim 8 , wherein the generating of the at least one field-level bounding box comprises, for each respective field-level bounding box, setting a height and width of the at least one field-level bounding box to cause the at least one field-level bounding box to surround each pixel predicted to belong to the respective field-level bounding box.
14. A system comprising:
a processor; and
a non-transitory memory in communication with the processor storing instructions that, when executed by the processor, cause the processor to perform processing comprising:
receiving an image;
generating at least one field-level bounding box indicating at least one field including at least a subset of one or more characters in the image, the generating of the at least one field-level bounding box comprising:
predicting, for each pixel, a field-level bounding box or a background region to which the pixel belongs, and
for each pixel belonging to a respective field-level bounding box, determining an offset of the pixel to a center of the respective field-level bounding box; and
overlaying the at least one field-level bounding box on the image to form a masked image including a plurality of characters and one or more predicted fields for at least a subset of the plurality of characters.
15. The system of claim 14 , wherein the processing further comprises:
extracting information contained within the one or more predicted fields of the masked image; and
populating one or more fillable fields corresponding to the one or more predicted fields with the information.
16. The system of claim 14 , wherein the processing further comprises identifying, by the processor, the one or more characters in the image using a machine learning (ML) model.
17. The system of claim 14 , wherein the predicting comprises:
applying a directed graph algorithm to identify a direction from the pixel towards the center; and
applying a non-maximum suppression algorithm to remove at least one redundancy.
18. The system of claim 14 , wherein the generating of the at least one field-level bounding box comprises, for each respective field-level bounding box, setting a height and width of the at least one field-level bounding box to cause the at least one field-level bounding box to surround each pixel predicted to belong to the respective field-level bounding box.
19. The system of claim 14 , wherein the overlaying comprises:
forming the at least one field-level bounding box into a bounding box mask, wherein the at least one field-level bounding box is located in one or more positions corresponding to the groups of characters in the image; and
overlaying the bounding box mask on the image.
20. The system of claim 14 , wherein the ML model comprises at least one mapping function used by the processor to perform the identifying and configured from training data supplied to the ML model.