Optical receipt processing
Techniques for providing improved optical character recognition (OCR) for receipts are discussed herein. Some embodiments may provide for a system including one or more servers configured to perform receipt image cleanup, logo identification, and text extraction. The image cleanup may include transforming image data of the receipt by using image parameters values that optimize the logo identification, and performing logo identification using a comparison of the image data with training logos associated with merchants. When a merchant is identified, a second image clean up may be performed by using image parameter values optimized for text extraction. A receipt structure may be used to categorize the extracted text. Improved OCR accuracy is also achieved by applying on format rules of the receipt structure to the extracted text.
1. A system, comprising:
one or more servers configured to:
receive, from a consumer device, image data associated with a receipt;
cause execution of a first image cleanup with respect to the image data associated with the receipt;
determine, based on the first image cleanup, a merchant logo associated with a merchant;
determine, based on the merchant logo, a receipt structure associated with the merchant;
cause execution of a second image cleanup with respect to the image data associated with the receipt;
extract, based on the second image cleanup, one or more portions of receipt text data from the image data;
determine a text match score based on a comparison of the one or more portions of receipt text data and one or more known keywords associated with the receipt structure related to the merchant;
determine whether the text match score satisfies the text match score threshold; and
in response to determining that the text match score satisfies the text match score threshold,
generate structured receipt data based on the one or more portions of receipt text data.
2. The system of claim 1 , wherein the first image cleanup comprises a first algorithmic transformation of one or more pixels comprised in the image data based on a first set of predefined image parameters, and
wherein the second image cleanup comprises a second algorithmic transformation of one or more pixels comprised in the image data based on a second set of predefined image parameters.
3. The system of claim 2 , wherein the first set of predefined image parameters and the second set of predefined image parameters comprise at least one of: contrast, brightness, filter size, offset, smoothing strategy, enhancement strategy, rotation, or skew.
4. The system of claim 2 , wherein the first image cleanup and the second image cleanup comprises performing one or more of: an image de-skew, an image resize, or an image de-warp.
5. The system of claim 1 , wherein the first image cleanup and the second image cleanup are performed iteratively based on a predefined number of iterations.
6. The system of claim 5 , wherein the one or more servers are further configured to:
in response to determining that the merchant logo associated with the merchant cannot be determined based on performing the first image cleanup for the predefined number of iterations:
cause the consumer device to capture second image data associated with the receipt; and
cause execution of the first image cleanup with respect to the second image data.
7. The system of claim 1 , wherein the text match score threshold is associated with a particular merchant, and wherein the text match score threshold is set for a predefined level of tolerance associated with the particular merchant.
8. The system of claim 1 , wherein the one or more portions of receipt text data can be categorized based on at least one of a field or a subject matter.
9. The system of claim 1 , wherein the structured receipt data comprises one or more portions of fine receipt data associated with one or more items, and wherein the one or more portions of fine receipt data comprise at least one of: an item name, an item stock keeping unit (SKU), an item price, or an item quantity.
10. A method comprising:
receiving, by one or more servers, image data associated with a receipt from a consumer device;
causing, by the one or more servers, execution of a first image cleanup with respect to the image data associated with the receipt;
determining, by the one or more servers, based on the first image cleanup, a merchant logo associated with a merchant;
determining, by the one or more servers, based on the merchant logo, a receipt structure associated with the merchant;
causing, by the one or more servers, execution of a second image cleanup with respect to the image data associated with the receipt;
extracting, by the one or more servers, based on the second image cleanup, one or more portions of receipt text data from the image data;
determining, by the one or more servers, a text match score based on a comparison of the one or more portions of receipt text data and one or more known keywords associated with the receipt structure related to the merchant;
determining, by the one or more servers, whether the text match score satisfies a text match score threshold; and
in response to determining that the text match score satisfies the text match score threshold,
generating, by the one or more servers, structured receipt data based on the one or more portions of receipt text data.
11. The method of claim 10 , wherein the first image cleanup comprises a first algorithmic transformation of one or more pixels comprised in the image data based on a first set of predefined image parameters, and
wherein the second image cleanup comprises a second algorithmic transformation of one or more pixels comprised in the image data based on a second set of predefined image parameters.
12. The method of claim 11 , wherein the first set of predefined image parameters and the second set of predefined image parameters comprise at least one of: contrast, brightness, filter size, offset, smoothing strategy, enhancement strategy, rotation, or skew.
13. The method of claim 11 , wherein the first image cleanup and the second image cleanup comprises performing one or more of: an image de-skew, an image resize, or an image de-warp.
14. The method of claim 10 , wherein the first image cleanup and the second image cleanup are performed iteratively based on a predefined number of iterations.
15. The method of claim 14 , the method further comprising:
in response to determining that the merchant logo associated with the merchant cannot be determined based on performing the first image cleanup for the predefined number of iterations:
causing, by the one or more servers, the consumer device to capture second image data associated with the receipt; and
causing, by the one or more servers, execution of the first image cleanup with respect to the second image data.
16. The method of claim 10 , wherein the text match score threshold is associated with a particular merchant, and wherein the text match score threshold is set for a predefined level of tolerance associated with the particular merchant.
17. The method of claim 10 , wherein the one or more portions of receipt text data can be categorized based on at least one of a field or a subject matter.
18. The method of claim 10 , wherein the structured receipt data comprises one or more portions of fine receipt data associated with one or more items, and wherein the one or more portions of fine receipt data comprise at least one of: an item name, an item stock keeping unit (SKU), an item price, or an item quantity.
19. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to:
receive, from a consumer device, image data associated with a receipt;
cause execution of a first image cleanup with respect to the image data associated with the receipt;
determine, based on the first image cleanup, a merchant logo associated with a merchant;
determine, based on the merchant logo, a receipt structure associated with the merchant;
cause execution of a second image cleanup with respect to the image data associated with the receipt;
extract, based on the second image cleanup, one or more portions of receipt text data from the image data;
determine a text match score based on a comparison of the one or more portions of receipt text data and one or more known keywords associated with the receipt structure related to the merchant;
determine whether the text match score satisfies the text match score threshold, and
in response to determining that the text match score satisfies the text match score threshold,
generate structured receipt data based on the one or more portions of receipt text data.
20. The computer program product of claim 19 , wherein the first image cleanup comprises a first algorithmic transformation of one or more pixels comprised in the image data based on a first set of predefined image parameters, and
wherein the second image cleanup comprises a second algorithmic transformation of one or more pixels comprised in the image data based on a second set of predefined image parameters.