Data extraction and duplicate detection
A system provides an end-to-end solution for invoice processing which includes reading invoices (both pdfs and images), extracting key relevant information from the face of invoices, organizing the relevant information in a structured template as a key-value pair, and comparing invoices based on the similarities between different invoice fields to identify potential duplicate invoices.
1. A method, comprising:
receiving, by a computer-based system, an invoice;
performing, by the computer-based system, optical character recognition on the invoice;
extracting, by the computer-based system, a plurality of key-value pairs from the invoice;
generating, by the computer-based system, a structured template comprising the plurality of key-value pairs;
forming, by the computer-based system, a feature vector;
determining, by the computer-based system using a duplicate model and based on the feature vector, that the invoice is a duplicate of a historic invoice in a historic invoice database;
receiving, by the computer-based system, an input to the duplicate model indicating an accuracy of the duplicate determination; and
modifying, by the computer-based system and based on the input, the duplicate model.
2. The method of claim 1 , wherein the forming the feature vector comprises concatenating similarity measures across different fields.
3. The method of claim 1 , further comprising executing, by the computer-based system, a word break algorithm to segment non-space separated words which exist in a reference dictionary.
4. The method of claim 1 , further comprising dividing, by the computer-based system, a plurality of bigrams in the invoice into categories consisting of (1) both words in a reference dictionary; (2) only first word in the reference dictionary; (3) only second word in the reference dictionary; and (4) neither word in the reference dictionary.
5. The method of claim 1 , further comprising performing, by the computer-based system, a table parsing operation on the invoice.
6. The method of claim 1 , further comprising saving, by the computer-based system, a value and location of a field in the invoice.
7. The method of claim 1 , further comprising creating, by the computer-based system, a lookup dictionary comprising description keywords.
8. A system comprising:
a processor; and
a tangible, non-transitory memory configured to communicate with the processor, the tangible, non-transitory memory having instructions stored thereon that, in response to execution by the processor, cause the processor to perform operations comprising:
receiving, by the processor, an invoice;
performing, by the processor, optical character recognition on the invoice;
extracting, by the processor, a plurality of key-value pairs from the invoice;
generating, by the processor, a structured template comprising the plurality of key-value pairs;
forming, by the processor, a feature vector;
determining, by the processor using a duplicate model and based on the feature vector, that the invoice is a duplicate of a historic invoice in a historic invoice database;
receiving, by the processor, an input to the duplicate model indicating an accuracy of the duplicate determination; and
modifying, by the processor and based on the input, the duplicate model.
9. The system of claim 8 , wherein the forming the feature vector comprises concatenating similarity measures across different fields.
10. The system of claim 8 , further comprising executing, by the processor, a word break algorithm to segment non-space separated words which exist in a reference dictionary.
11. The system of claim 8 , further comprising dividing, by the processor, a plurality of bigrams in the invoice into categories consisting of (1) both words in a reference dictionary; (2) only first word in the reference dictionary; (3) only second word in the reference dictionary; and (4) neither word in the reference dictionary.
12. The system of claim 8 , further comprising performing, by the processor, a table parsing operation on the invoice.
13. The system of claim 8 , further comprising saving, by the processor, a value and location of a field in the invoice.
14. The system of claim 8 , further comprising creating, by the processor, a lookup dictionary comprising description keywords.
15. An article of manufacture including a non-transitory, tangible computer readable storage medium having instructions stored thereon that, in response to execution by a computer-based system, cause the computer-based system to perform operations comprising:
receiving, by the computer-based system, an invoice;
performing, by the computer-based system, optical character recognition on the invoice;
extracting, by the computer-based system, a plurality of key-value pairs from the invoice;
generating, by the computer-based system, a structured template comprising the plurality of key-value pairs;
forming, by the computer-based system, a feature vector;
determining, by the computer-based system using a duplicate model and based on the feature vector, that the invoice is a duplicate of a historic invoice in a historic invoice database;
receiving, by the computer-based system, an input to the duplicate model indicating an accuracy of the duplicate determination; and
modifying, by the computer-based system and based on the input, the duplicate model.
16. The article of manufacture of claim 15 , wherein the forming the feature vector comprises concatenating similarity measures across different fields.
17. The article of manufacture of claim 15 , further comprising executing, by the computer-based system, a word break algorithm to segment non-space separated words which exist in a reference dictionary.
18. The article of manufacture of claim 15 , further comprising dividing, by the computer-based system, a plurality of bigrams in the invoice into categories consisting of (1) both words in a reference dictionary; (2) only first word in the reference dictionary; (3) only second word in the reference dictionary; and (4) neither word in the reference dictionary.
19. The article of manufacture of claim 15 , further comprising performing a table parsing operation on the invoice.
20. The article of manufacture of claim 15 , further comprising saving, by the computer-based system, a value and location of a field in the invoice.