Method and system for preprocessing digital documents for data extraction
Techniques described herein relate to a method for performing preprocessing of documents for data extraction. The method includes obtaining a document preprocessing request; in response to obtaining a document preprocessing request: obtaining a document associated with the document preprocessing request; performing data preparation on the document to generate an updated document; generating a document type prediction using the updated document and a document type prediction model; identifying a data extraction service of a plurality of data extraction services associated with the document type prediction; and initiating further processing of the document to perform data extraction using the identified data extraction service.
1 . A method for performing preprocessing of documents for data extraction, comprising:
identifying a document type prediction model generation event;
in response to identifying the document type prediction model generation event:
obtaining training documents associated with the document type prediction model generation event;
performing data preparation on the training documents to generate updated training documents, wherein the updated training documents comprise augmented training documents and new training documents, wherein:
generating the augmented training documents comprises performing blurring, rotating, flipping, adding noise to, or shape shifting to the training documents, and
generating the new training documents comprises using linear interpolations of training document image tensors between at least two different document image tensor representations of corresponding ones of the training documents;
generating the document type prediction model using the updated training documents;
after the generating, obtaining a document preprocessing request;
in response to obtaining a document preprocessing request:
obtaining a document associated with the document preprocessing request;
performing data preparation on the document to generate an updated document;
generating a document type prediction using the updated document and a document type prediction model;
identifying a data extraction service of a plurality of data extraction services associated with the document type prediction; and
initiating further processing of the document to perform data extraction using the identified data extraction service.
2 . The method of claim 1 , further comprising:
after initiating the further processing of the document using the identified data extraction service:
obtaining a second document associated with the document preprocessing request;
performing data preparation on the second document to generate a second updated document;
generating a second document type prediction using the second updating document and the document type prediction model;
identifying a second data extraction service of the plurality of data extraction services associated with the second document type prediction; and
initiating further processing of the second document using the second identified data extraction service.
3 . The method of claim 2 , wherein the document type prediction specifies a first document type and the second document type prediction specifies a second document type.
4 . The method of claim 3 , wherein:
the data extraction service is configured to perform data extraction for documents of the first document type; and
the second data extraction service is configured to perform data extraction for documents of the second document type.
5 . The method of claim 1 , wherein performing data preparation on the document to generate the updated document comprises:
performing augmentations on the document; and
performing transformations on the document.
6 . The method of claim 1 , wherein performing data preparation on the training documents to generate the updated training documents comprises:
performing augmentations on the training documents; and
performing transformations on the training documents.
7 . A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for performing preprocessing of documents for data extraction, the method comprising:
identifying a document type prediction model generation event;
in response to identifying the document type prediction model generation event:
obtaining training documents associated with the document type prediction model generation event;
performing data preparation on the training documents to generate updated training documents, wherein the updated training documents comprise augmented training documents and new training documents, wherein:
generating the augmented training documents comprises performing blurring, rotating, flipping, adding noise to, or shape shifting to the training documents, and
generating the new training documents comprises using linear interpolations of training document image tensors between at least two different document image tensor representations of corresponding ones of the training documents;
generating the document type prediction model using the updated training documents;
after the generating, obtaining a document preprocessing request;
in response to obtaining a document preprocessing request:
obtaining a document associated with the document preprocessing request;
performing data preparation on the document to generate an updated document;
generating a document type prediction using the updated document and a document type prediction model;
identifying a data extraction service of a plurality of data extraction services associated with the document type prediction; and
initiating further processing of the document to perform data extraction using the identified data extraction service.
8 . The non-transitory computer readable medium of claim 7 , further comprising:
after initiating the further processing of the document using the identified data extraction service:
obtaining a second document associated with the document preprocessing request;
performing data preparation on the second document to generate a second updated document;
generating a second document type prediction using the second updating document and the document type prediction model;
identifying a second data extraction service of the plurality of data extraction services associated with the second document type prediction; and
initiating further processing of the second document using the second identified data extraction service.
9 . The non-transitory computer readable medium of claim 8 , wherein the document type prediction specifies a first document type and the second document type prediction specifies a second document type.
10 . The non-transitory computer readable medium of claim 9 wherein:
the data extraction service is configured to perform data extraction for documents of the first document type; and
the second data extraction service is configured to perform data extraction for documents of the second document type.
11 . The non-transitory computer readable medium of claim 7 , wherein performing data preparation on the document to generate the updated document comprises:
performing augmentations on the document; and
performing transformations on the document.
12 . The non-transitory computer readable medium of claim 7 , wherein performing data preparation on the training documents to generate the updated training documents comprises:
performing augmentations on the training documents; and
performing transformations on the training documents.
13 . A system for performing data extraction for documents comprises:
a plurality of clients; and
a document preprocessing engine configured to:
identify a document type prediction model generation event;
in response to identifying the document type prediction model generation event:
obtain training documents associated with the document type prediction model generation event;
perform data preparation on the training documents to generate updated training documents, wherein the updated training documents comprise augmented training documents and new training documents, wherein:
generating the augmented training documents comprises performing blurring, rotating, flipping, adding noise to, or shape shifting to the training documents, and
generating the new training documents comprises using linear interpolations of training document image tensors between at least two different document image tensor representations of corresponding ones of the training documents;
generate the document type prediction model using the updated training documents;
after the generating, obtain a document preprocessing request from a client of the plurality of clients;
in response to obtaining a document preprocessing request:
obtain a document associated with the document preprocessing request;
perform data preparation on the document to generate an updated document;
generate a document type prediction using the updated document and a document type prediction model;
identify a data extraction service of a plurality of data extraction services associated with the document type prediction; and
initiate further processing of the document to perform data extraction using the identified data extraction service.
14 . The system of claim 13 , wherein the document preprocessing engine is further configured to:
after initiating the further processing of the document using the identified data extraction service:
obtain a second document associated with the document preprocessing request;
perform data preparation on the second document to generate a second updated document;
generate a second document type prediction using the second updating document and the document type prediction model;
identify a second data extraction service of the plurality of data extraction services associated with the second document type prediction; and
initiate further processing of the second document using the second identified data extraction service.
15 . The system of claim 14 , wherein the document type prediction specifies a first document type and the second document type prediction specifies a second document type.
16 . The system of claim 15 , wherein:
the data extraction service is configured to perform data extraction for documents of the first document type; and
the second data extraction service is configured to perform data extraction for documents of the second document type.
17 . The system of claim 13 , wherein performing data preparation on the document to generate the updated document comprises:
performing augmentations on the document; and
performing transformations on the document.