IP Library Granted Patent US 11,423,075
Granted Patent B2
US 11,423,075 · App. 16/569,139 · Granted Aug 23, 2022

Product auditing in point-of-sale images

Inventors: Diego García Morate (Valladolid, ES); Antonio Hurtado García (Valladolid, ES); Jorge Martin Martin (Valladolid, ES); Miguel Garcia Carranza (Valladolid, ES)
Assignee: Nielsen Consumer LLC
G06F16/51G06F16/5854G06Q10/087
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,423,075
App. No.
16/569,139
Granted
Aug 23, 2022
Kind
B2
Abstract

Examples methods, apparatus/systems and articles of manufacture for auditing point-of-sale images are disclosed herein. Example methods disclosed herein include comparing a region of interest of an image displayed via a user interface with a plurality of reference product images stored in a database to identify a plurality of candidate product images from the plurality of reference product images as potential matches to a first product depicted in the image. For example, the candidate product images are associated with respective confidence levels indicating respective likelihoods of matching the first product. Disclosed example methods also include displaying, via the user interface, the candidate product images simultaneously with the image in a manner based on the respective confidence levels, and automatically selecting a first one of the candidate product images as matching the first product based on the respective confidence levels.

Claims (38)

1. An apparatus comprising:

a user interface;

machine readable instructions; and

processor circuitry to execute the machine readable instructions to:

cause the user interface to display a segmentation of an image depicting a plurality of products, the segmentation generated by an automatic image recognition application, the segmentation including the image and a first frame around a first location of a first product in the image identified by the automatic image recognition application;

in response to a user selection of the first frame:

generate a first preview window including first product information associated with the first product;

cause the user interface to display the first preview window over a section of the image such that the first preview window is displayed simultaneously with and adjacent to the first frame; and

update a first product record associated with the first product based on a modification made, via the user interface, to the first product information displayed in the first preview window.

2. The apparatus of claim 1 , wherein the first product information includes at least one of an image of the first product, a name of the first product, a view or position of the first product, a layout of the first product, an indication of whether the first product is in stock, an indication of whether the first product is on a pallet or a price of the first product.

3. The apparatus of claim 2 , wherein the layout of the first product includes a number of rows and columns of the first product.

4. The apparatus of claim 2 , wherein the modification is a change to at least one of the name of the first product, the view or position of the first product, the layout of the first product, the indication of whether the first product is in stock, the indication of whether the first product is on a pallet or the price of the first product.

5. The apparatus of claim 1 , wherein the processor circuitry is to cause the user interface to display, via the first preview window, a first plurality of candidate reference products identified by the automatic image recognition application as potential matches to the first product.

6. The apparatus of claim 5 , wherein the modification is based on a user selection of a first one of the first plurality of candidate reference products, and the processor circuitry is to update the first product record by assigning the first one of the first plurality of candidate reference products as matching the first product.

7. The apparatus of claim 5 , wherein the segmentation further includes a second frame overlaying the image and defining a second location of a second product identified by the automatic image recognition application, the processor circuitry is further to generate, in response to a user selection of the second frame, a second preview window including second product information associated with the second product, the second preview window to be displayed via the user interface simultaneously with the second frame.

8. The apparatus of claim 7 , wherein the processor circuitry is to cause the user interface to display, via the second preview window, a second plurality of candidate reference products identified by the automatic image recognition application as potential matches to the second product.

9. A non-transitory machine readable storage medium comprising instructions that, when executed, cause a machine to at least:

display a segmentation of a point-of-sale image depicting a plurality of products, the segmentation generated by an automatic image recognition application, the segmentation including the image and a first frame overlaying the image and surrounding a first location of a first product in the image identified by the automatic image recognition application;

display, in response to a user selection of the first frame, a first quick preview window over a portion of the image such that the first quick preview window is displayed simultaneously with and adjacent the first frame with the first product, the first quick preview window including first product information associated with the first product; and

update a first product record associated with the first product based on a modification to the first product information as received in the first quick preview window.

10. The non-transitory machine readable storage medium of claim 9 , wherein the first product information includes at least one of an image of the first product, a name of the first product, a view or position of the first product, a layout of the first product, an indication of whether the first product is in stock, an indication of whether the first product is on a pallet or a price of the first product.

11. The non-transitory machine readable storage medium of claim 9 , wherein the instructions, when executed, further cause the machine to display, via the first quick preview window, a first plurality of candidate reference products identified by the automatic image recognition application as potential matches to the first product.

12. The non-transitory machine readable storage medium of claim 11 , wherein the modification is based on a user selection of a first one of the first plurality of candidate reference products, and the instructions, when executed, cause the machine to update the first product record by assigning the first one of the first plurality of candidate reference products as matching the first product.

13. The non-transitory machine readable storage medium of claim 11 , wherein the segmentation further includes a second frame overlaying the image and defining a second location of a second product identified by the automatic image recognition application, and the instructions, when executed, further cause the machine to:

display, in response to a user selection of the second frame, a second quick preview window simultaneously with the second frame, the second quick preview window including second product information associated with the second product.

14. The non-transitory machine readable storage medium of claim 13 , wherein the instructions, when executed, further cause the machine to:

display, via the second quick preview window, a second plurality of candidate reference products identified by the automatic image recognition application as potential matches to the second product.

15. A method comprising:

displaying, via a user interface, a segmentation of an image depicting a plurality of products, the segmentation generated by an automatic image recognition application, the segmentation including the image and a first frame overlaying the image and being around a first product in the image identified by the automatic image recognition application;

displaying, in response to a user selection of the first frame, a first preview window over a section of the image adjacent to the first frame such that the first preview window is display simultaneously with and adjacent the first frame and the first product in the first frame via the user interface, the first preview window including first product information associated with the first product; and

updating, with a processor, a first product record associated with the first product based on a modification made, via the user interface, to the first product information displayed in the first preview window.

16. The method of claim 15 , wherein the first product information includes at least one of an image of the first product, a name of the first product, a view or position of the first product, a layout of the first product, an indication of whether the first product is in stock, an indication of whether the first product is on a pallet or a price of the first product.

17. The method of claim 15 , further including displaying, via the first preview window, a first plurality of candidate reference products identified by the automatic image recognition application as potential matches to the first product.

18. The method of claim 17 , wherein the modification is based on a user selection of a first one of the first plurality of candidate reference products, and the updating of the first product record includes assigning the first one of the first plurality of candidate reference products as matching the first product.

19. The method of claim 17 , wherein the segmentation further includes a second frame overlaying the image and defining a second location of a second product identified by the automatic image recognition application, and further including:

displaying, in response to a user selection of the second frame, a second preview window simultaneously with the second frame, the second preview window including second product information associated with the second product.

20. The method of claim 19 , further including:

displaying, via the second preview window, a second plurality of candidate reference products identified by the automatic image recognition application as potential matches to the second product.

Assignments (8)
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
SECURITY INTEREST Recorded Mar 25, 2021
From: NIELSEN CONSUMER LLC; BYZZER INC.
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT AND COLLATERAL AGENT
Reel/Frame 055742/0719 →
PARTIAL RELEASE OF SECURITY INTEREST Recorded Mar 10, 2021
From: CITIBANK, N.A.
To: NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN CONSUMER LLC
Reel/Frame 055557/0949 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 17, 2021
From: THE NIELSEN COMPANY (US), LLC
To: NIELSEN CONSUMER LLC
Reel/Frame 055325/0353 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 17, 2020
From: MORATE, DIEGO GARCÍA; GARCÍA, ANTONIO HURTADO; MARTIN, JORGE MARTIN; CARRANZA, MIGUEL GARCIA
To: THE NIELSEN COMPANY (US), LLC
Reel/Frame 051641/0194 →
Continuity (2)
Continuation 14787935
Related Publication 20200004773A1 · Jan 2, 2020
Cited By (3)
US 12,189,915 US 12,211,161 US 12,430,856