Method of automatically extracting information of a predefined type from a document
Method and system of automatically extracting information of a predefined type from a document is provided. The method comprises using an object detection algorithm to identify at least one segment of the document that is likely to comprise the information of the predefined type. The method further comprises building at least one bounding box corresponding to the at least one segment and if the bounding box is likely to comprise the information of the predefined type extracting the information comprised by the bounding box from the at least one bounding box.
1. A method comprising:
identifying, by an object detection algorithm, at least one segment of a document that is likely to comprise information of a predefined type;
building at least one bounding box corresponding to the at least one segment;
identifying that the at least one bounding box likely comprises the information of the predefined type; and
extracting, by a character identification algorithm, the information of the predefined type from the at least one bounding box based on identifying, by a multilayer neural network, the information of the predefined type based on characteristics of the information of the predefined type, wherein the neural network includes:
a first layer configured to differentiate between empty regions and non-empty regions of the document and to identify basic patterns present on the document, and
a second layer configured to identify shapes that are more complex compared to the basic patterns present on the document.
2. The method of claim 1 wherein the characteristics of the information of the predefined type comprise a number format and at least one of a comma or a decimal point.
3. The method of claim 1 wherein the multilayer neural network is compatible with a decision layer, and the decision layer is configured to detect at least one of (i) a location of the bounding box, (ii) a height and a width of a bounding box, and (iii) a classification score indicating a classification of a detected character.
4. The method of claim 1 wherein a convolutional multilayer neural network is used by the object detection algorithm.
5. The method of claim 1 wherein a fully-convolutional neural network is used by the object detection algorithm and/or the character identification algorithm.
6. The method of claim 1 further comprising:
training the neural network with a plurality of documents in a training activity to correctly extract the information of the predefined type.
7. The method of claim 1 wherein a probability value is assigned to the at least one bounding box, and the probability value is indicative of the probability that a certain bounding box contains the information of the predefined type.
8. The method of claim 1 further comprising:
identifying a character by the character identification algorithm,
wherein a probability value is assigned to the character, and the probability value is indicative of the probability that the identified character is identical with a character actually comprised by the information of the predefined type.
9. The method of claim 1 further comprising:
assigning a probability value assigned to the at least one bounding box; and
assigning probability values to characters within the at least one bounding box in order to provide a combined confidence score.
10. The method of claim 1 wherein the document is digitally scanned from a paper-based document, and the information of the predefined type is at least one of a creation date, a total amount, an arrival/departure date, a VAT-ID, a receipt id, and an invoice number.
11. The method of claim 1 wherein the document is a paper-based receipt or a paper-based invoice.
12. A system comprising:
a computing device; and
a computer-readable storage medium comprising a set of instructions that upon execution by the computing device cause the system to:
identify, by an object detection algorithm, at least one segment of a document that is likely to comprise information of a predefined type;
build at least one bounding box corresponding to the at least one segment;
identify that the at least one bounding box likely comprises the information of the predefined type; and
extract, by a character identification algorithm, the information of the predefined type from the at least one bounding box based on identifying, by a multilayer neural network, the information of the predefined type based on characteristics of the information of the predefined type, wherein the neural network includes:
a first layer configured to differentiate between empty regions and non-empty regions of the document and to identify basic patterns present on the document, and
a second layer configured to identify shapes that are more complex compared to the basic patterns present on the document.
13. The system of claim 12 wherein the characteristics of the information of the predefined type comprise a number format and at least one of a comma or a decimal point.
14. The system of claim 12 wherein the multilayer neural network is compatible with a decision layer, and the decision layer is configured to detect at least one of (i) a location of the bounding box, (ii) a height and a width of a bounding box, and (iii) a classification score indicating a classification of a detected character.
15. The system of claim 12 wherein the set of instructions, upon execution by the computing device, further cause the system to:
train the neural network with a plurality of documents in a training activity to correctly extract the information of the predefined type.
16. The system of claim 12 wherein a probability value is assigned to the at least one bounding box, and the probability value is indicative of the probability that a certain bounding box contains the information of the predefined type.
17. The system of claim 12 wherein the set of instructions, upon execution by the computing device, further cause the system to:
identify a character by the character identification algorithm,
wherein a probability value is assigned to the character, and the probability value is indicative of the probability that the identified character is identical with a character actually comprised by the information of the predefined type.
18. The system of claim 12 wherein the set of instructions, upon execution by the computing device, further cause the system to:
assign a probability value assigned to the at least one bounding box; and
assign probability values to characters within the at least one bounding box in order to provide a combined confidence score.
19. The system of claim 12 wherein the document is a paper-based receipt or a paper-based invoice.
20. A non-transitory computer-readable storage medium comprising computer-readable instructions that upon execution by a processor of a computing device cause the computing device to:
identify, by an object detection algorithm, at least one segment of a document that is likely to comprise information of a predefined type;
build at least one bounding box corresponding to the at least one segment;
identify that the at least one bounding box likely comprises the information of the predefined type; and
extract, by a character identification algorithm, the information of the predefined type from the at least one bounding box based on identifying, by a multilayer neural network, the information of the predefined type based on characteristics of the information of the predefined type, wherein the neural network includes:
a first layer configured to differentiate between empty regions and non-empty regions of the document and to identify basic patterns present on the document, and
a second layer configured to identify shapes that are more complex compared to the basic patterns present on the document.