Data normalization and extraction system
A data ingestion system normalizes ingested documents and extracts data based on a template that is applied to the documents. In an aspect, the system accesses a document of a document type and determines a template to apply to the document. The system normalizes the document, extracts data values from the document based at least in part on the template, and generates structured data based at least partly on the extracted data.
1. A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
accessing a digital image of a document comprising a plurality of rows of pixels;
for each of a plurality of rotations of the document:
generating an array of row values, each row value associated with a different row of pixels of the document; and
calculating an overall value for the rotation of the document based at least on relationships between the row values;
identifying the lowest overall value; and
rotating content of the document by a rotation angle associated with the identified lowest overall value.
2. The computerized method of claim 1 , further comprising:
generating a user interface configured to display one or more templates, wherein each template is associated with a document type;
receiving a user selection defining one or more template input fields and one or more template graphical portions associated with the one or more template input fields; and
generating one or more templates based at least partly on the user selection, wherein each template is associated with a document type.
3. The computerized method of claim 2 , wherein the user selection defining one or more template input fields and one or more template graphical portions comprises clicking and dragging an area of the template.
4. The computerized method of claim 2 , further comprising:
extracting data values from the document for each of the template input fields identified in the template based at least partly on the associated graphical portions of the document associated with the template input fields.
5. The computerized method of claim 4 , wherein said extracting data values comprises comparing a first position of a template data field to a second position of a graphical portion associated with the document.
6. The computerized method of claim 4 , wherein data values are extracted if a first portion of a graphical element of the document extends beyond a template graphical portion as long as a remaining second portion of the graphical element overlaps the template graphical portion by more than a predetermined overlap percentage.
7. The computerized method of claim 6 , wherein the predetermined overlap percentage is adjustable by a user.
8. The computerized method of claim 4 , wherein said extracting data values comprises using optical character recognition techniques to recognize hand-written text.
9. The computerized method of claim 1 , wherein the rotating is based at least partly on the rotation angle and an axis point.
10. The computerized method of claim 1 , wherein the array of row values is a one-dimensional array of values having a size of a total number of pixel rows in the digital image.
11. The computerized method of claim 1 , further comprising:
shifting content of the document to a predefined horizontal or vertical position.
12. The computerized method of claim 11 , wherein said shifting content is based at least partly on a magnitude and a direction.
13. The computerized method of claim 12 , wherein the magnitude of shifting is determined based at least partly on detecting a first column of pixels associated with a pixel density exceeding a threshold value.
14. The computerized method of claim 12 , wherein the magnitude and direction of shifting are determined based at least partly on a pixel density of a column of pixels in the document.
15. The computerized method of claim 11 , wherein the predefined horizontal and vertical positions correspond to an x-y coordinate system associated with the document.
16. A computing system comprising:
a hardware computer processor;
a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
accessing a digital image of a document comprising a plurality of rows of pixels;
for each of a plurality of rotations of the document:
generating an array of row values, each row value associated with a different row of pixels of the document; and
calculating an overall value for the rotation of the document based at least on relationships between the row values;
identifying the lowest overall value; and
rotating content of the document by a rotation angle associated with the identified lowest overall value.
17. A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by a hardware computer processor to cause a computing system to perform operations comprising:
accessing a digital image of a document comprising a plurality of rows of pixels;
for each of a plurality of rotations of the document:
generating an array of row values, each row value associated with a different row of pixels of the document; and
calculating an overall value for the rotation of the document based at least on relationships between the row values;
identifying the lowest overall value; and
rotating content of the document by a rotation angle associated with the identified lowest overall value.