System for information extraction from form-like documents
The present disclosure is directed to extracting text from form-like documents. In particular, a computing system can obtain an image of a document that contains a plurality of portions of text. The computing system can extract one or more candidate text portions for each field type included in a target schema. The computing system can generate a respective input feature vector for each candidate for the field type. The computing system can generate a respective candidate embedding for the candidate text portion. The computing system can determine a respective score for each candidate text portion for the field type based at least in part on the respective candidate embedding for the candidate text portion. The computing system can assign one or more of the candidate text portions to the field type based on the respective scores.
1. A computer-implemented method for extracting information from images of structured documents, the method comprising:
obtaining, by a computing system comprising one or more computing devices, an image of a document, wherein the image of the document includes one or more portions of text;
determining, by the computing system, a document type associated with the document;
accessing, by the computing system, a target schema associated with the document type, the target schema including one or more field types;
providing, by the computing system, the image of the document as input to a machine-learned model; and
receiving, by the computing system and from the machine-learned model, model output, the model output associating a respective portion of the document with a respective field type in the one or more field types, wherein the model output includes, for a plurality of candidate portions in the one or more portions of text, a score assigned to a respective candidate portion based on a degree to which it is associated with the respective field type in the one or more field types.
2. The computer-implemented method of claim 1 , wherein the respective portion of the document is selected to be associated with the respective field type based on one or more scores associated with one or more candidate portions of the document.
3. The computer-implemented method of claim 1 , wherein the score for the respective candidate portion is determined based on a similarity metric between the respective candidate portion and one or more field characteristics associated with the field type.
4. The computer-implemented method of claim 3 , wherein the similarity metric comprises a cosine similarity metric.
5. The computer-implemented method of claim 1 , wherein providing, by the computing system, the image of the document as input to a machine-learned model further comprises:
extracting, by the computing system from the image of the document, one or more candidate portions for each of one or more field types included in a target schema; and
providing, by the computing system, the extracted one or more candidate portions to the machine-learned model.
6. The computer-implemented method of claim 5 , wherein providing, by the computing system, the extracted one or more candidate portions to the machine-learned model further comprises:
providing, by the computing system, for a respective candidate portion, data describing a respective position of one or more neighbor text portions that are proximate to the respective candidate portion and data describing a relative normalized position of the one or more neighbor text portions relative to the respective candidate portion.
7. The computer-implemented method of claim 6 , wherein providing, by the computing system, for a respective candidate portion, data describing the respective position of one or more neighbor portions that are proximate to the respective candidate portion comprises:
defining, by the computing system, a respective neighborhood zone for each candidate portion; and
identifying, by the computing system, the one or more neighbor portions for each candidate portion based at least in part on the respective neighborhood zone for each candidate portion and the respective positions of the one or more neighbor portions.
8. The computer-implemented method of claim 7 , wherein defining, by the computing system, the respective neighborhood zone for each candidate portion comprises, for each candidate portion:
defining, by the computing system, the respective neighborhood zone to extend from a position of the candidate portion leftwards to a margin of the document and to extend from the position of the candidate portion upwards a threshold amount of the document.
9. The computer-implemented method of claim 5 , wherein providing, by the computing system, the extracted one or more candidate portions to the machine-learned model further comprises:
providing, by the computing system, for a respective candidate portion, data describing an absolute position of the respective candidate portion.
10. The computer-implemented method of claim 5 , wherein providing, by the computing system, the extracted one or more candidate portions to the machine-learned model further comprises:
providing, by the computing system, for a respective candidate portion, data describing text contained in the respective candidate portion.
11. A computing system for extracting information from images of structured documents, the system comprising:
one or more processors; and
a non-transitory computer-readable memory that stores instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining an image of a document, wherein the image of the document includes one or more portions of text;
determining a document type associated with the document;
accessing a target schema associated with the document type, the target schema including one or more field types;
providing the image of the document as input to a machine-learned model; and
receiving, from the machine-learned model, model output, the model output associating a respective portion of the document with a respective field type in the one or more field types, wherein the model output includes, for a plurality of candidate portions in the one or more portions of text, a score assigned to a respective candidate portion based on a degree to which it is associated with the respective field type in the one or more field types.
12. The computing system of claim 11 , wherein the respective portion of the document is selected to be associated with the respective field type based on one or more scores associated with one or more candidate portions of the document.
13. A non-transitory computer-readable medium storing instruction that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining an image of a document, wherein the image of the document includes one or more portions of text;
determining a document type associated with the document;
accessing a target schema associated with the document type, the target schema including one or more field types;
providing the image of the document as input to a machine-learned model; and
receiving, from the machine-learned model, model output, the model output associating a respective portion of the document with a respective field type in the one or more field types, wherein the model output includes, for a plurality of candidate portions in the one or more portions of text, a score assigned to a respective candidate portion based on a degree to which it is associated with the respective field type in the one or more field types.
14. The non-transitory computer-readable medium of claim 13 , wherein the respective portion of the document is selected to be associated with the respective field type based on one or more scores associated with one or more candidate portions of the document.