Multi-layer approach to improving generation of field extraction models
A computer-implemented process for generating cluster templates used for creating extraction models includes the following operations. A plurality of training files associated with a selected class are received. An automated visual analysis is performed on each of the plurality of training files. An automated contextual analysis is performed on each of the plurality of training files. A first clustering of the plurality of training files into a first plurality of clusters using results from the automated visual analysis is performed. A second clustering of one of the plurality of clusters into a second plurality of clusters is performed using results from the automated contextual analysis. Cluster templates for the first and second plurality of clusters are generated.
1. A computer-implemented method for generating cluster templates used for creating extraction models, comprising:
receiving a plurality of training files associated with a selected class;
performing an automated visual analysis on each of the plurality of training files;
performing an automated contextual analysis on each of the plurality of training files;
performing a first clustering of the plurality of training files into a first plurality of clusters using results from the automated visual analysis;
performing a second clustering of one of the first plurality of clusters into a second plurality of clusters using results from the automated contextual analysis; and
generating cluster templates for the first and second plurality of clusters, wherein
the first and the second plurality of clusters are clusters of the plurality of training files.
2. The method of claim 1 , further comprising:
receiving, for a first training file within a particular one of the first and second plurality of clusters, annotations;
automatically propagating the annotations to each of a plurality of other training files within the particular one of the first and second plurality of clusters; and
generating a cluster annotation template for the particular one of the first and second plurality of clusters using the annotations.
3. The method of claim 2 , further comprising:
receiving, from a user, an instruction to modify the annotations; and
modifying the cluster annotation template based upon the instruction.
4. The method of claim 2 , further comprising:
generating field extraction models for each of the first and second plurality of clusters using the cluster templates and cluster annotation templates.
5. The method of claim 1 , wherein
the automated visual analysis includes:
generating an image hash for each of the plurality of training files, and
extracting structural information for each of the plurality of training files.
6. The method of claim 5 , wherein
the image hash includes generating a shingle.
7. The method of claim 1 , wherein
the automated contextual analysis includes generating a Term Frequency-Inverse Document Frequency vector for each of the plurality of training files.
8. The method of claim 1 , further comprising:
receiving a document for data extraction;
identifying a classification of the document;
selecting a field extraction model based upon the classification; and
extracting data from the document using the field extraction model.
9. A computer hardware system for generating cluster templates used for creating extraction models, comprising:
a hardware processor configured to perform the following executable operations:
receiving a plurality of training files associated with a selected class;
performing an automated visual analysis on each of the plurality of training files;
performing an automated contextual analysis on each of the plurality of training files;
performing a first clustering of the plurality of training files into a first plurality of clusters using results from the automated visual analysis;
performing a second clustering of one of the first plurality of clusters into a second plurality of clusters using results from the automated contextual analysis; and
generating the cluster templates for the first and second plurality of clusters, wherein
the first and the second plurality of clusters are clusters of the plurality of training files.
10. The system of claim 9 , wherein the hardware processor is further configured to perform the following:
receiving, for a first training file within a particular one of the first and second plurality of clusters, annotations;
automatically propagating the annotations to each of a plurality of other training files within the particular one of the first and second plurality of clusters; and
generating a cluster annotation template for the particular one of the first and second plurality of clusters using the annotations.
11. The system of claim 10 , wherein the hardware processor is further configured to perform the following:
receiving, from a user, an instruction to modify the annotations; and
modifying the cluster annotation template based upon the instruction.
12. The system of claim 10 , wherein the hardware processor is further configured to perform the following:
generating field extraction models for each of the first and second plurality of clusters using the cluster templates and cluster annotation templates.
13. The system of claim 9 , wherein
the automated visual analysis includes:
generating an image hash for each of the plurality of training files, and
extracting structural information for each of the plurality of training files.
14. The system of claim 13 , wherein
the image hash includes generating a shingle.
15. The system of claim 9 , wherein
the automated contextual analysis includes generating a Term Frequency-Inverse Document Frequency vector for each of the plurality of training files.
16. The system of claim 9 , wherein the hardware processor is further configured to perform the following:
receiving a document for data extraction;
identifying a classification of the document;
selecting a field extraction model based upon the classification; and
extracting data from the document using the field extraction model.
17. A computer program product, comprising:
a computer readable storage medium having stored therein program code for generating cluster templates used for creating extraction models,
the program code, which when executed by a computer hardware system, cause the computer hardware system to perform:
receiving a plurality of training files associated with a selected class;
performing an automated visual analysis on each of the plurality of training files;
performing an automated contextual analysis on each of the plurality of training files;
performing a first clustering of the plurality of training files into a first plurality of clusters using results from the automated visual analysis;
performing a second clustering of one of the first plurality of clusters into a second plurality of clusters using results from the automated contextual analysis; and
generating the cluster templates for the first and second plurality of clusters, wherein
the first and the second plurality of clusters are clusters of the plurality of training files.
18. The computer program product of claim 17 , wherein the computer hardware system is further configured to perform the following:
receiving, for a first training file within a particular one of the first and second plurality of clusters, annotations;
automatically propagating the annotations to each of a plurality of other training files within the particular one of the first and second plurality of clusters;
generating a cluster annotation template for the particular one of the first and second plurality of clusters using the annotations;
receiving, from a user, an instruction to modify the annotations;
modifying the cluster annotation template based upon the instruction; and
generating field extraction models for each of the first and second plurality of clusters using the cluster templates and cluster annotation templates.
19. The computer program product of claim 17 , wherein
the automated visual analysis includes:
generating an image hash for each of the plurality of training files, and
extracting structural information for each of the plurality of training files,
the image hash includes generating a shingle, and
the automated contextual analysis includes generating a Term Frequency-Inverse Document Frequency vector for each of the plurality of training files.
20. The computer program product of claim 17 , wherein the program code further causes the computer hardware system to perform:
receiving a document for data extraction;
identifying a classification of the document;
selecting a field extraction model based upon the classification; and
extracting data from the document using the field extraction model.