Optical receipt processing
Techniques for providing improved optical character recognition (OCR) for receipts are discussed herein. Some embodiments may provide for a system including one or more servers configured to perform receipt image cleanup, logo identification, and text extraction. The image cleanup may include transforming image data of the receipt by using image parameters values that optimize the logo identification, and performing logo identification using a comparison of the image data with training logos associated with merchants. When a merchant is identified, a second image clean up may be performed by using image parameter values optimized for text extraction. A receipt structure may be used to categorize the extracted text. Improved OCR accuracy is also achieved by applying on format rules of the receipt structure to the extracted text.
1. A system comprising:
one or more servers; and
a consumer device, the consumer device configured to:
capture image data;
execute at least one image cleanup process with respect to the image data, wherein the at least one image cleanup process comprises:
iteratively adjusting at least one image parameter value to programmatically identify receipt image data from the image data; and
extracting the receipt image data from the image data; and
output the extracted receipt image data to the one or more servers, wherein the one or more servers are configured to:
determine a merchant associated with the receipt image data, wherein determining the merchant associate with the receipt image data is based at least on logo detection, the logo detection comprising:
identifying a merchant logo within the receipt image data;
accessing training logo image data defining training logos associated with a plurality of merchants;
determining logo match scores between the identified merchant logo and the training logos; and
determining whether the logo match scores satisfy a logo match score threshold.
2. The system of claim 1 , wherein iteratively adjusting at least one image parameter value to programmatically identify receipt image data from the image data comprises:
setting one or more image parameters to predefined values according to a first set of predefined image parameters; and
iteratively adjusting the at least one image parameter value of the first set of predefined image parameters to distinguish the receipt image data from background image data.
3. The system of claim 2 , wherein the first set of predefined image parameters comprises at least one of contrast, brightness, filter size, offset, smoothing strategy, enhancement strategy, rotation, or skew.
4. The system of claim 3 , wherein iteratively adjusting the at least one image parameter value comprises:
identifying one or more receipt edges; and
performing one or more of an image de-skew, an image resize, or an image de-warp.
5. The system of claim 3 , wherein iteratively adjusting the at least one image parameter value comprises iteratively adjusting the contrast of the image data until edges of the receipt image data are identified within the image data.
6. The system of claim 5 , wherein the consumer device is further configured to:
perform a pixel-by-pixel analysis of each iteratively adjusted image to identify at least one of a color or a shape of a receipt corresponding to the receipt image data in the image data.
7. The system of claim 1 , wherein iteratively adjusting image parameter values is based on a predefined number of iterations.
8. The system of claim 7 , wherein the consumer device is further configured to:
capture second image data in an instance wherein one or more receipt edges are not identified in the image data subsequent to the predefined number of iterations.
9. The system of claim 1 , wherein determining the merchant associated with the receipt image data further comprises:
determining a merchant identifier associated with the receipt image data based on the logo detection.
10. The system of claim 1 , wherein the one or more servers are further configured to:
determine a receipt structure associated with the merchant, wherein the receipt structure defines one or more receipt elements and one or more locations respectively associated with the one or more receipt elements.
11. The system of claim 10 , wherein the one or more receipt elements include one or more transaction receipt elements, category receipt elements, or item receipt elements.
12. The system of claim 10 , wherein the one or more receipt elements include one or more of a stock keeping unit identifier receipt element, an item type receipt element, an item brand receipt element, a unit size receipt element, a unit count receipt element, a per unit price receipt element, or a total item price receipt element.
13. The system of claim 1 , wherein determining the merchant associated with the receipt image data further comprises:
receiving merchant identifier input via the consumer device.
14. The system of claim 1 , wherein determining the merchant associated with the receipt image data further comprises:
determining a consumer device location of the consumer device; and
associating the consumer device location with a known merchant location.
15. The system of claim 1 , wherein the one or more servers are further configured to perform text extraction on the receipt image data to generate receipt text data.
16. A computer-implemented method comprising:
causing capture of image data;
causing execution of at least one image cleanup process with respect to the image data, wherein the at least one image cleanup process comprises:
iteratively adjusting at least one image parameter value to programmatically identify receipt image data from the image data; and
extracting the receipt image data from the image data;
causing transmission of the extracted receipt image data to one or more servers; and
determining a merchant associated with the receipt image data, wherein determining the merchant associate with the receipt image data is based at least on logo detection, the logo detection comprising:
identifying a merchant logo within the receipt image data;
accessing training logo image data defining training logos associated with a plurality of merchants;
determining logo match scores between the identified merchant logo and the training logos; and
determining whether the logo match scores satisfy a logo match score threshold.
17. The computer-implemented method of claim 16 , further comprising:
determining a receipt structure associated with the merchant, wherein the receipt structure defines one or more receipt elements and one or more locations respectively associated with the one or more receipt elements.
18. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to:
cause capture of image data;
cause execution of at least one image cleanup process with respect to the image data, wherein the at least one image cleanup process comprises:
iteratively adjusting at least one image parameter value to programmatically identify receipt image data from the image data; and
extracting the receipt image data from the image data;
cause transmission of the extracted receipt image data to one or more servers; and
determine a merchant associated with the receipt image data, wherein determining the merchant associate with the receipt image data is based at least on logo detection, the logo detection comprising:
identifying a merchant logo within the receipt image data;
accessing training logo image data defining training logos associated with a plurality of merchants;
determining logo match scores between the identified merchant logo and the training logos; and
determining whether the logo match scores satisfy a logo match score threshold.
19. The computer program product of claim 18 , the computer-executable program code instructions comprising program code instructions further configured to:
determine a receipt structure associated with the merchant, wherein the receipt structure defines one or more receipt elements and one or more locations respectively associated with the one or more receipt elements.
20. The system of claim 7 , wherein the consumer device is further configured to:
send a request to a display of the consumer device to capture second image data in an instance wherein receipt edges are not identified subsequent to the predefined number of iterations.