Systems and methods for automatic data extraction from document images
Described systems and methods allow the automatic extraction of structured information from images of structured text documents such as invoices and receipts. Some embodiments employ optical character recognition (OCR) technology to extract individual text tokens (e.g., words) and token bounding boxes from a document image. A feature vector of each text token comprises a first part determined according to a character content of the text token, and a second part determined according to an image content of the token's bounding box. A neural network classifier produces a label indicative of a type of information (e.g. “billing address”, “due date”, etc.) carried by each text token. In some embodiments, documents are linearized by ordering text tokens in a sequence according to a reading order of a natural language (e.g., English, Arabic) in which the respective document is formulated. Token feature vectors are fed to the classifier in the order indicated by the token sequence.
1. A method comprising employing at least one hardware processor of a computer system to:
linearize a text content of a document image comprising an encoding of an image of a structured paper document, the structured paper document partitioned into a plurality of fields and having a plurality of text tokens distributed among the plurality of fields, each field of the plurality of fields having a distinct field type characterizing a distinct category of information represented by text tokens located within the each field, wherein linearizing the text content of the document image comprises:
dividing the plurality of text tokens into a plurality of text lines of the structured paper document, and
concatenating the plurality of text lines to form an ordered token sequence;
assemble an ordered input according to an order of the ordered token sequence, the ordered input comprising, for each text token of the ordered token sequence, a feature vector characterizing a respective text token as a whole and determined according to a character content of the respective text token and further according to a pixel content of a region of the document image showing the respective text token; and
feed the ordered input to a neural network configured to determine a field type of a field containing the respective text token according to the ordered input.
2. The method of claim 1 , wherein constructing a text line of the plurality of text lines comprises:
determining a line guide of the text line, the line guide comprising a line passing through a set of vertices of a bounding box of a text token belonging to the text line; and
determining whether another text token of the plurality of text tokens belongs to the text line according to a distance between a bounding box of the other text token and the line guide.
3. The method of claim 2 , further comprising, in response to determining whether the other text token belongs to the text line, when yes, updating the line guide according to a set of vertices of the bounding box of the other text token.
4. The method of claim 1 , comprising ordering the token sequence according to a natural language that the structured paper document is formulated in.
5. The method of claim 1 , wherein arranging the plurality of text tokens into the token sequence comprises:
determining whether a first text token of the plurality of text tokens is located to the left of a second text token within the structured paper document; and
in response, when yes, ordering the token sequence so that the first text token precedes the second text token within the ordered token sequence.
6. The method of claim 1 , wherein arranging the plurality of text tokens into the token sequence comprises:
determining whether a first text token of the plurality of text tokens is located closer to the top of the structured paper document than a second text token; and
in response, when yes, ordering of the token sequence so that the first text token precedes the second text token within the ordered token sequence.
7. The method of claim 1 , wherein the structured paper document comprises an item selected from a group consisting of an invoice and a receipt.
8. The method of claim 1 , wherein the structured paper document is crumpled.
9. The method of claim 1 , wherein the ordered token sequence spans the entire structured paper document.
10. The method of claim 1 , wherein the neural network is configured to determine the field type according to the feature vector characterizing the respective text token, and further according to feature vectors characterizing other text tokens preceding and following the respective text token within the ordered token sequence.
11. A computing system comprising at least one hardware processor configured to execute a line segmentation engine and a token classifier connected to the line segmentation engine, wherein:
the line segmentation engine is configured to linearize a text content of a document image comprising an encoding of an image of a structured paper document, the structured paper document partitioned into a plurality of fields and having a plurality of text tokens distributed among the plurality of fields, each field of the plurality of fields having a distinct field type characterizing a distinct category of information represented by text tokens located within the each field, wherein linearizing the text content of the document image comprises:
dividing the plurality of text tokens into a plurality of text lines of the structured paper document, and
concatenating the plurality of text lines to form the ordered token sequence; and
the token classifier is configured to:
assemble an ordered input according to an order of the ordered token sequence, the ordered input comprising, for each text token of the ordered token sequence, a feature vector characterizing a respective text token as a whole and determined according to a character content of the respective text token and further according to a pixel content of a region of the document image showing the respective text token, and
feed the ordered input to a neural network configured to determine a field type of a field containing the respective text token according to the ordered input.
12. The computing system of claim 11 , wherein constructing a text line of the plurality of text lines comprises:
determining a line guide of a text line the line guide comprising a line passing through a set of vertices of a bounding box of a text token belonging to the text line; and
determining whether another text token of the plurality of text tokens belongs to the text line according to a distance between a bounding box of the other text token and the line guide.
13. The computing system of claim 12 , wherein constructing the text line further comprises, in response to determining whether the other text token belongs to the text line, when yes, updating the line guide according to a set of vertices of the bounding box of the other text token.
14. The computing system of claim 11 , wherein arranging the plurality of text tokens into the token sequence comprises ordering the token sequence according to a natural language that the structured paper document is formulated in.
15. The computing system of claim 11 , wherein arranging the plurality of text tokens into the token sequence comprises:
determining whether a first text token of the plurality of text tokens is located to the left of a second text token within the structured paper document; and
in response, when yes, ordering the token sequence so that the first text token precedes the second text token within the ordered token sequence.
16. The computing system of claim 11 , wherein arranging the plurality of text tokens into the token sequence comprises:
determining whether a first text token of the plurality of text tokens is located closer to the top of the structured paper document than a second text token; and
in response, when yes, ordering the token sequence so that the first text token precedes the second text token within the ordered token sequence.
17. The computing system of claim 11 , wherein the structured paper document comprises an item selected from a group consisting of an invoice and a receipt.
18. The computing system of claim 11 , wherein the structured paper document is crumpled.
19. The computing system of claim 11 , wherein the ordered token sequence spans the entire structured paper document.
20. The computing system of claim 11 , wherein the neural network is configured to determine the field type according to the feature vector characterizing the respective text token, and further according to feature vectors characterizing other text tokens preceding and following the respective text token within the ordered token sequence.
21. A non-transitory computer-readable medium storing instructions which, when executed by at least one hardware processor of a computing system, cause the computing system to form a line segmentation engine, a text feature extractor, an image feature extractor, and a token classifier connected to the text feature extractor, the image feature extractor, and the line segmentation engine, wherein:
the line segmentation engine is configured to linearize a text content of a document image comprising an encoding of an image of a structured paper document, the structured paper document partitioned into a plurality of fields and having a plurality of text tokens distributed among the plurality of fields, each field of the plurality of fields having a distinct field type characterizing a distinct category of information represented by text tokens located within the each field, wherein linearizing the text content of the document image comprises:
dividing the plurality of text tokens into a plurality of text lines of the structured paper document, and
concatenating the plurality of text lines to form the ordered token sequence; and
the token classifier is configured to:
assemble an ordered input according to an order of the ordered token sequence, the ordered input comprising, for each text token of the ordered token sequence, a feature vector characterizing a respective text token as a whole and determined according to a character content of the respective text token and further according to a pixel content of a region of the document image showing the respective text token, and
feed the ordered input to a neural network configured to determine a field type of a field containing the respective text token according to the ordered input.