Granted Patent B2

US 12,229,741 · App. 17/710,660 · Granted Feb 18, 2025

Inventors: Jose Javier Yebes Torres (Valladolid, ES); Aditi Sinha (Weehawken, NY); Christine Lebrun (Paris, FR); Fabio Oppini (Milan, IT); Atul Bansal (New York, NY); Mukul Kumar (New York, NY); Vignesh Chandramouli (New York, NY); Filipa Sousa (New York, NY); Gisella Mercaldi (Oxford, GB)

Assignee: Nielsen Consumer LLC

G06Q20/201G06K7/1413G06V30/414G06V30/42

Abstract

Claims (71)

1. An apparatus comprising:

a first dictionary including images of product descriptions and barcodes;

interface circuitry;

machine readable instructions; and

at least one processor circuit to be programmed by the machine readable instructions to:

train a region detection machine learning model based on (a) the images of the product descriptions and barcodes, (b) ground truth annotations, and (c) intersection over union (IoU) comparisons of bounding boxes associated with the images;

based on satisfying a threshold IoU value of the region detection machine learning model, obtain purchase details and barcodes corresponding to a receipt, the purchase details including a first product description;

generate a first search query that includes the first product description, a list of barcodes corresponding to the barcodes, and a store identifier associated with the receipt;

execute a first search against the first dictionary using the first search query to identify a first barcode from the list of barcodes that corresponds to the first product description;

generate a second search query when the first search does not identify the first barcode, the second search query to include the first product description, the list of barcodes, and the store identifier;

execute a second search against a second dictionary based on the second search query to identify the first barcode;

associate the first barcode and the first product description; and

add the association to the first dictionary.

2. The apparatus of claim 1 , wherein the list of barcodes is a list of unique barcodes, one or more of the at least one processor circuit is to remove duplicate ones of the barcodes to generate the unique list of barcodes.

3. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to execute the first search against the first dictionary using the first search query by:

identifying first candidate barcodes in the first dictionary that correspond to the list of barcodes in the first search query;

identifying second candidate barcodes from the first candidate barcodes that correspond to the store identifier; and

identifying a first one of the second candidate barcodes, the first one of the second candidates barcodes to that include the first product description.

4. The apparatus of claim 1 , wherein the second dictionary is a products datastore that includes a plurality of products and corresponding attributes, ones of the attributes to include, at least, a respective second barcode and a respective second product description.

5. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to execute the second search against the second dictionary using the second search query includes by:

identifying second barcodes in the second dictionary that correspond to the list of barcodes in the second search query;

comparing second product descriptions associated with the second barcodes identified in the second dictionary to the first product description in the second search query;

generating a similarity value for ones of the second product descriptions associated with the second barcodes identified in the second dictionary based on the comparison; and

selecting a candidate barcode from the list of barcodes based on the similarity value for the ones of the second product descriptions associated with the second barcodes identified in the second dictionary, the similarity value of the candidate barcode to reach a threshold value.

6. The apparatus of claim 5 , wherein the execution of the second search reveals a plurality of the second barcodes as having a respective second product description that receives a similarity score that reaches the threshold value, one or more of the at least one processor circuit is to select one of the plurality of the second barcodes that includes a similarity value with a largest value.

7. The apparatus of claim 5 , wherein one or more of the at least one processor circuit is to:

identify a match between the first product description and the candidate barcode;

generate an association between the first product description and the candidate barcode; and

add the association to the second dictionary.

8. The apparatus of claim 5 , wherein the execution of the second search does not identify a candidate barcode in the second dictionary that includes a respective second product description that receives a similarity score above the threshold value, one or more of the at least one processor circuit is to:

adjust the second search query to remove the store identifier; and

execute a third search against the second dictionary using the adjusted second search query to identify the candidate barcode from the list of barcodes that corresponds to the first product description.

9. The apparatus of claim 8 , wherein, after executing searches against the first dictionary and the second dictionary for the receipt product descriptions and the receipt barcodes corresponding to the receipt, one or more of the at least one processor circuit is to add unassociated receipt product descriptions and unassociated receipt barcodes to a list of unassociated products.

10. The apparatus of claim 9 , further including a third dictionary, wherein one or more of the at least one processor circuit is to:

generate a fourth search query that includes the unassociated receipt product descriptions from the list of unassociated products, the unassociated receipt barcodes from the list of unassociated products, and the store identifier corresponding to the receipt; and

execute a fourth search against the third dictionary using the fourth search query.

11. The apparatus of claim 10 , wherein the third dictionary is a database that includes records of previous searches corresponding to a plurality of processed receipts that did not yield associations between at least one product description and at least one barcode, ones of the records including at least unassociated product description and at least one unassociated barcode that were not matched.

12. The apparatus of claim 11 , wherein one or more of the at least one processor circuit is to execute the fourth search against the third dictionary using the fourth search query by:

comparing the unassociated receipt product descriptions and the unassociated receipt barcodes of the fourth search query to the processed receipts;

identifying a first processed receipt that includes one of the unassociated receipt product descriptions and one of the unassociated receipt barcodes;

in response to identifying the first processed receipt, identifying a match between the one of the unassociated receipt product descriptions and the one of the unassociated receipt barcodes; and

in response to not identifying the first processed receipt, adding the one of the unassociated receipt product descriptions and the one of the unassociated receipt barcodes corresponding to the receipt to as a record to a previous jobs database.

13. At least one non-transitory computer readable storage medium comprising instructions to cause at least one processor circuit to at least:

train a region detection machine learning model based on (a) images of product descriptions and barcodes in a first dictionary, (b) ground truth annotations, and (c) intersection over union (IoU) comparisons of bounding boxes associated with the images;

based on satisfying a threshold IoU value of the region detection machine learning model, obtain purchase details and barcodes corresponding to a receipt, the purchase details including a first product description;

generate a first search query that includes the first product description, a list of barcodes corresponding to the barcodes, and a store identifier associated with the receipt;

execute a first search against the first dictionary using the first search query to identify a first barcode from the list of barcodes that corresponds to the first product description;

generate a second search query when the first search does not identify the first barcode, the second search query to include the first product description, the list of barcodes, and the store identifier;

execute a second search against a second dictionary based on the second search query to identify the first barcode;

associate the first barcode and the product description; and

add the association to the first dictionary.

14. The at least one non-transitory computer readable storage medium as defined in claim 13 , wherein the list of barcodes is a list of unique barcodes, the instructions are to cause the processor circuit to remove duplicate ones of the barcodes to generate the unique list of barcodes.

15. The at least one non-transitory computer readable storage medium as defined in claim 13 , wherein the instructions are to cause one or more of the at least one processor circuit to execute the search against the first dictionary using the first search query by:

identifying first candidate barcodes in the first dictionary that correspond to the list of barcodes in the first search query;

identifying second candidate barcodes from the first candidate barcodes that correspond to the store identifier; and

identifying a first one of the second candidate barcodes that includes the first product description.

16. The at least one non-transitory computer readable storage medium as defined in claim 13 , wherein the second dictionary is a products datastore that includes a plurality of products and corresponding attributes, ones of the attributes to include, at least, a respective second barcode and a respective second product description.

17. The at least one non-transitory computer readable storage medium as defined in claim 13 , wherein the instructions are to cause one or more of the at least one processor circuit to execute the second search against the second dictionary using the second search query by:

identifying second barcodes in the second dictionary that correspond to the list of barcodes in the second search query;

comparing second product descriptions associated with the second barcodes identified in the second dictionary to the first product description in the second search query;

generating a similarity value for ones of the second product descriptions associated with the second barcodes identified in the second dictionary based on the comparison; and

selecting a candidate barcode from the second barcodes based on the similarity value for the ones of the second product descriptions associated with the second barcodes identified in the second dictionary, the similarity value of the candidate barcode to reach a threshold value.

18. A method comprising:

training, by executing machine readable instructions with at least one processor circuit, a region detection machine learning model based on (a) images of product descriptions and barcodes, (b) ground truth annotations, and (c) intersection over union (IoU) comparisons of bounding boxes associated with the images;

based on satisfying a threshold IoU value of the region detection machine learning model, obtaining, by executing machine learning instructions with the at least one processor circuit, purchase details and barcodes corresponding to a receipt, the purchase details including a first product description;

generating, by executing machine readable instructions with the least one processor circuit, a first search query that includes the first product description, a list of barcodes corresponding to the barcodes, and a store identifier associated with the receipt;

executing, by executing machine readable instructions with the at least one processor circuit, a first search against the first dictionary using the first search query to identify a barcode from the list of barcodes that corresponds to the first product description;

generating, by executing machine readable instructions with the at least one processor circuit, a second search query when the first search does not identify the barcode, the second search query to include the first product description, the list of barcodes, and the store identifier;

executing, by executing machine readable instructions with the at least one processor circuit, a second search against a second dictionary based on the second search query to identify the barcode;

associating, by executing machine readable instructions with the at least one processor circuit, the barcode and the product description; and

adding the association to the first dictionary.

Assignments (7)

Continuity (2)

References Cited (355)