Granted Patent B2

US 12,327,425 · App. 17/710,649 · Granted Jun 10, 2025

Inventors: Jose Javier Yebes Torres (Valladolid, ES); Aditi Sinha (Weehawken, NY); Christine Lebrun (Paris, FR); Fabio Oppini (Milan, IT); Atul Bansal (New York, NY); Mukul Kumar (New York, NY); Vignesh Chandramouli (New York, NY); Filipa Sousa (New York, NY); Gisella Mercaldi (Oxford, GB)

Assignee: Nielsen Consumer LLC

G06V30/416G06Q30/0283G06V30/147G06V30/15G06V30/413G06V30/414

Abstract

Claims (55)

1. An apparatus comprising:

interface circuitry to obtain an image of a receipt;

machine-readable instructions; and

at least one processor circuit to be programmed by the machine-readable instructions to at least:

cause generation of a first cropped image and a second cropped image based on the image of the receipt, the first and second cropped images based on regions of the receipt detected in the image;

apply a first mask to the first cropped image to generate first bounding boxes, the first bounding boxes corresponding to rows of the receipt;

apply a second mask to the second cropped image to generate second bounding boxes, the second bounding boxes corresponding to columns of the receipt;

generate a structure of the receipt by mapping (a) words detected by an optical character recognition (OCR) engine to corresponding first bounding boxes based on a mapping criterion, and (b) ones of the words to respective ones of the second bounding boxes based on the mapping criterion;

cause classification of the columns based on the ones of the words mapped to the respective ones of the second bounding boxes;

extract purchase information from the structured receipt based on the classifications, the purchase information corresponding to products listed in the receipt, the products including a first product; and

generate promotion information based on the first product being associated with a promotion by:

identifying a first price associated with the first product, the first price corresponding to a first column, the first column classified as a price column;

identifying a second price associated with the first product, the second price corresponding to the promotion; and

determining a third price associated with the first product based on the first price and the second price.

2. The apparatus of claim 1 , wherein the regions include (1) a receipt region, and (2) a purchase region, the purchase region corresponding to an area of the receipt that includes information about the products listed in the receipt.

3. The apparatus of claim 2 , wherein the first cropped image is based on the receipt region and the second cropped image is based on the purchase region.

4. The apparatus of claim 1 , wherein the first mask causes identification of first pixels of the first cropped image classified as belonging to a first class and second pixels of the first cropped image classified as belonging to a second class.

5. The apparatus of claim 4 , wherein the first class is corresponds to a text line and the second class is corresponds to a background.

6. The apparatus of claim 1 , wherein the second mask identifies of first pixels of the second cropped image classified as belonging to a first class and second pixels of the second cropped image classified as belonging to a second class.

7. The apparatus of claim 6 , wherein the first class is corresponds to a column and the second class corresponds to another class.

8. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to merge a first one of the first bounding boxes and a second one of the first bounding boxes based on a row connection criterion, the first one and the second one of the first bounding boxes to satisfy a length criterion.

9. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to merge a first one of the second bounding boxes and a second one of the second bound boxes based on a column connection criterion.

10. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to identify first words of the ones of the words assigned to the respective ones of the second bounding boxes that include a regular expression, the regular expression to correspond to a targeted fact.

11. The apparatus of claim 10 , wherein the targeted fact includes at least one of a product description, a quantity, or a price.

12. The apparatus of claim 1 , wherein the purchase information includes purchase details and promotion information, and wherein one or more of the at least one processor circuit is to;

identify first ones of the words assigned to the first bounding boxes that do not correspond to a respective classification of a respective bounding box; and

remove the first ones of the words from the first bounding boxes prior to extracting the purchase details.

13. The apparatus of claim 12 , wherein the purchase details include, for respective ones of the products in the receipt, a respective product description, a respective price, and a respective quantity.

14. The apparatus of claim 13 , wherein one or more of the at least one processor circuitry circuit is to cause decoding of the purchase details to decoding circuitry.

15. The apparatus of claim 1 , wherein the first price is an original price of the first product, the second price is a discount amount, and the third price is a purchase price for the first product.

16. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to apply a trained object detection model to the image to detect the regions.

17. At least one non-transitory computer readable storage medium comprising instructions that, when executed, to cause at least one processor circuit to at least:

cause generation of a first receipt region and a second receipt region based on an image of a receipt, the first and second receipt regions based on regions detected in the image;

apply a first mask to the first receipt region to generate first bounding boxes, the first bounding boxes corresponding to rows of the receipt;

apply a second mask to the second of the receipt region to generate second bounding boxes, the second bounding boxes corresponding to columns of the receipt;

form a structure of the receipt by (a) assigning words detected by an optical character recognition (OCR) engine to corresponding first bounding boxes based on a mapping criterion and (b) assigning ones of the words to respective ones of the second bounding boxes based on the mapping criterion;

cause classification of the the columns based on ones of the words assigned to the respective ones of the second bounding boxes;

extract purchase information from the structured receipt based on the classifications, the purchase information to include items listed in the receipt, the items to include a first item associated with a promotion; and

generate promotion information for the first item by:

identifying a first price associated with the first item, the first price corresponding identified from a first column classified as a price column;

identifying a second price associated with the first item, the second price corresponding to the promotion; and

determining a third price associated with the first item based on the first price and the second price.

18. The at least one non-transitory computer readable storage medium as defined in claim 17 , wherein the instructions, are to cause one or more of the at least one processor circuit to merge a first one of the first bounding boxes and a second one of the first bounding boxes based on a row connection criterion, the first one and the second one of the first bounding boxes to satisfy a length criterion.

19. The at least one non-transitory computer readable storage medium as defined in claim 17 , wherein the instructions cause one or more of the at least one processor circuit to apply a trained object detection model to the image to detect the regions.

20. An apparatus comprising:

means for generating row bounding boxes to generate row bounding boxes on an image based on a first mask, the first mask applied to a first region detected in the image, the row bounding boxes corresponding to rows of a receipt;

means for generating column bounding boxes to generate column bounding boxes on the image based on a second mask, the second mask to applied to a second region detected in the image, the column bounding boxes corresponding to columns of the receipt; and

means for extracting purchase information to:

generate a data structure to represent the receipt by (a) mapping words extracted from the image to respective ones of the row bounding boxes based on a mapping criterion and (b) mapping ones of the words to respective ones of the column bounding boxes based on the mapping criterion;

cause classification of the columns based on respective ones of the words mapped to the respective ones of the column bounding boxes, the columns including a price column;

extract purchase data from the data structure based on the classifications, the purchase data including a first product indicated in the receipt, the first product associated with a promotion; and

generate promotion data for the first product by:

identifying a first price associated with the first product, the first price identified from the price column;

identifying a second price associated with the first product, the second price corresponding to the promotion; and

determining a third price associated with the first product based on the first price and the second price.

Assignments (5)

Continuity (2)

References Cited (359)