Systems and methods for machine learning based content extraction from document images
A method and apparatus for recognizing and extracting data from a form depicted within an image of a document are described. The method may include receiving the image of the document, the image depicting the form and data contained one the form. The method may also include transforming the image of the document to a set of one or more key, value pairs by processing the image of the document with a sequence of two or more trained machine learning based image analysis processes, wherein keys are relevant to forms of the type depicted in the form, and wherein each value is associated with a key. The method may also include generating a data output that comprises the set of key, value pairs for textual data recognized and extracted from the form depicted in the image.
1. A method for recognizing and extracting data from a form depicted within an image of a document, the method comprising:
receiving, by a computer processing system, the image of the document, the image depicting the form and data contained on the form;
transforming, by the computer processing system, the image of the document to a set of one or more key, value pairs by processing the image of the document with a sequence of two or more trained machine learning based image analysis processes, wherein keys are relevant to forms of the type depicted in the form, and wherein each value is associated with a key, wherein the two or more trained machine learning based image analysis processes comprise a plurality of different machine learning based image analysis processes performed within a document processing pipeline; and
generating, by the computer processing system, a data output that comprises the set of key, value pairs for textual data recognized and extracted from the form depicted in the image.
2. The method of claim 1 , wherein a description of a layout and a structure of the form depicted in the image of the document is unknown to the computer processing system.
3. The method of claim 1 , wherein the method further comprises:
performing a first machine learning based image analysis process to predict a rotation of the form depicted in the image of the document based on features extracted by a first machine learning based image analysis of the received image of the document; and
generating a rotated document image that rotates the form depicted in the image of the document based on the predicted rotation.
4. The method of claim 3 , wherein the form is rotated to a vertical orientation as depicted in the rotated document image.
5. The method of claim 3 , wherein the method further comprises:
performing a second machine learning based image analysis process to detect and locate text segments on the form depicted in the rotated document image based on features extracted by a second machine learning based image analysis of the rotated document image.
6. The method of claim 5 , wherein the method further comprises:
performing a third machine learning based image analysis process to extract features in the detected and located text segments based using a third machine learning based image analysis of the detected and located text segments;
performing a fourth machine learning based image analysis process to generate text character probabilities for each text character within the detected and located text segments based using a fourth machine learning based image analysis of the features extracted from the detected and located text segments by the third machine learning based image analysis;
selecting a most likely text character for each text character within the detected and located text segments based on the generated text character probabilities; and
generating textual content for each detected and located text segment based on the selection of text characters for said detected and located text segment.
7. The method of claim 6 , further comprising:
performing a fifth machine learning based image analysis process to classify textual content for detected and located text segments using a fifth machine learning based image analysis of at least a location associated with a segment and text content contained within said segment.
8. The method of claim 1 , wherein each of the plurality of different machine learning based image analysis processes comprises a machine learning analysis system executed by the computer processing system, and wherein each machine learning analysis system corresponding to each of the plurality of different machine learning based image analysis processes is trained using different training data relevant to a phase in the document processing pipeline in which machine learning image analysis is being performed.
9. The method of claim 1 , wherein the form depicted within the image of the document comprises one of a medical form, an insurance form, or a loan application form.
10. A non-transitory computer readable storage medium including instructions that, when executed by a computer processing system, cause the computer processing system to perform operations for recognizing and extracting data from a form depicted within an image of a document, the operations comprising:
receiving the image of the document, the image depicting the form and data contained one the form, and wherein the form depicted in the image is unstructured;
transforming the image of the document to a set of one or more key, value pairs by processing the image of the document with a sequence of two or more trained machine learning based image analysis processes, wherein keys are relevant to forms of the type depicted in the image, and wherein each value is associated with a key, wherein the two or more trained machine learning based image analysis processes comprise a plurality of different machine learning based image analysis processes performed within a document processing pipeline; and
generating a data output that comprises the set of key, value pairs for textual data recognized and extracted from the form depicted in the image.
11. The non-transitory computer readable storage medium of claim 10 , wherein a description of a layout and a structure of the form depicted in the image of the document is unknown to the computer processing system.
12. The non-transitory computer readable storage medium of claim 10 , wherein the operations further comprise:
performing a first machine learning based image analysis process to predict a rotation of the form depicted in the image of the document based on features extracted by a first machine learning based image analysis of the received image of the document; and
generating a rotated document image that rotates the form depicted in the image of the document based on the predicted rotation.
13. The non-transitory computer readable storage medium of claim 12 , wherein the form is rotated to a vertical orientation as depicted in the rotated document image.
14. The non-transitory computer readable storage medium of claim 12 , wherein the operations further comprise:
performing a second machine learning based image analysis process to detect and locate text segments on the form depicted in the rotated document image based on features extracted by a second machine learning based image analysis of the rotated document image.
15. The non-transitory computer readable storage medium of claim 14 , wherein the method further comprises:
performing a third machine learning based image analysis process to extract features in the detected and located text segments based using a third machine learning based image analysis of the detected and located text segments;
performing a fourth machine learning based image analysis process to generate text character probabilities for each text character within the detected and located text segments based using a fourth machine learning based image analysis of the features extracted from the detected and located text segments by the third machine learning based image analysis;
selecting a most likely text character for each text character within the detected and located text segments based on the generated text character probabilities; and
generating textual content for each detected and located text segment based on the selection of text characters for said each detected and located text segment.
16. The non-transitory computer readable storage medium of claim 15 , wherein the operations further comprise:
performing a fifth machine learning based image analysis process to classify textual content for detected and located text segments using a fifth machine learning based image analysis of at least a location associated with a segment and text content contained within said segment.
17. The non-transitory computer readable storage medium of claim 10 , wherein each of the plurality of different machine learning based image analysis processes comprises of techniques for machine learning analysis system executed by the computer processing system, and wherein each machine learning analysis system corresponding to each of the plurality of different machine learning based image analysis processes is trained using different training data relevant to a phase in the document processing pipeline.
18. The non-transitory computer readable storage medium of claim 10 , wherein the form depicted within the image of the document comprises one of a medical form, an insurance form, or a loan application form.
19. A system, comprising:
a network interface that receives an image of a document, the image depicting a form and data contained on the form, and wherein the form depicted in the image is unstructured;
a memory that stores the image of the document; and
a processor coupled with the memory configured to access the image of the document and further configured to:
transform the image of the document to a set of one or more key, value pairs by processing the image of the document with a sequence of two or more trained machine learning based image analysis processes, wherein keys are relevant to forms of the type depicted in the image, and wherein each value is associated with a key, wherein the two or more trained machine learning based image analysis processes comprise a plurality of different machine learning based image analysis processes performed within a document processing pipeline, and
generate a data output that comprises the set of key, value pairs for textual data recognized and extracted from the form depicted in the image.
20. The system of claim 19 , wherein a description of a layout and a structure of the form depicted in the image of the document is unknown to the computer processing system.