IP Library Granted Patent US 12,056,756
Granted Patent B2
US 12,056,756 · App. 17/464,925 · Granted Aug 6, 2024

Withholding low confidence notification due to planogram incompliance

Inventors: Yair Adato (Kfar Shmuel, IL); Ran Taig (Beer Sheva, IL)
Assignee: Trax Technology Solutions Pte Ltd.
G06Q30/0639G06F18/21G06Q10/06312G06Q10/06315G06Q10/06375G06Q10/087G06Q20/208G06Q30/0201G06Q30/0207G06Q30/0223G06Q30/0242G06Q30/0261G06Q30/0641G06Q30/0643G06T7/001G06T7/70G06V20/00G06V20/13G06V20/20G06V20/52H02J7/00036H02J7/0013H02J7/0048H02J7/0068H02J9/06H04N7/18H04N7/181H04N23/51H04N23/54H04N23/65G06T2207/30232G06V30/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,056,756
App. No.
17/464,925
Granted
Aug 6, 2024
Kind
B2
Abstract

A system for automatically identifying situations to withhold planogram incompliance notifications in a retail store is disclosed. The system has a processor that receives image data depicting products displayed on a store shelf from image sensors mounted in the retail store. The processor identifies a product type associated with the displayed products and determines a placement of products of that product type on the shelf. The processor accesses a planogram describing a desired placement of products and identifies, based on the planogram, a discrepancy between the determined placement of products and the desired placement of product. The processor determines whether a notice-override condition exists. If a notice-override condition exists, the processor, withholds issuance of a user-notification associated with the identified discrepancy. If a notice-override condition does not exist, the processor issues the user-notification associated with the identified discrepancy.

Claims (64)

1. A system for processing images captured in a retail store and automatically identifying situations to withhold planogram incompliance notifications, the system comprising:

at least one processor configured to:

receive first image data from a plurality of image sensors mounted in the retail store, wherein the first image data depicts a plurality of products displayed on at least one store shelf;

analyze the first image data using a machine learning model to identify at least one product type associated with the plurality of products displayed on at least one store shelf and to determine a placement of products of the at least one product type on the at least one store shelf;

access, in a database, at least one planogram describing a desired placement of products of the at least one product type on shelves of a retail store;

identify, using a discrepancy detection module, a discrepancy between the determined placement of products associated with the identified at least one product type and the desired placement of products associated with the identified at least one product type determined from the at least one planogram;

receive second image data captured by the plurality of image sensors prior to capturing the first image data;

analyze the second image data to identify a past placement of the products of the at least one product type;

determine, using a notice module, whether a notice-override condition exists relative to the identified at least one product type based on the identified discrepancy and the past placement of the products of the at least one product type;

if a notice-override condition exists relative to the identified at least one product type, withhold issuance of a user-notification associated with the identified discrepancy between the determined placement of products and the desired placement of products; and

if a notice-override condition is not determined to exist relative to the identified at least one product type, issue a user-notification associated with the identified discrepancy between the determined placement of products and the desired placement of products, using a network interface.

2. The system of claim 1 , wherein the at least one processor is configured to identify the discrepancy in the product placement when the desired placement of products indicates that products from a first product type are to be placed on the at least one shelf and the determined placement of products indicates that products from a second product type are placed on the at least one shelf.

3. The system of claim 2 , wherein the at least one processor is further configured to retrieve information indicative of at least one past instance in which products from the first product type were erroneously identified as products from the second product type.

4. The system of claim 3 , wherein the at least one processor is configured to:

determine, based on the at least one past instance, existence of a notice-override condition associated with the identified at least one product type, wherein the notice-override condition includes an erroneous identification of the products placed on the at least one shelf as products from the second product type; and

withhold issuance of a user-notification due to erroneous identification of the products placed on the at least one shelf as products from the second product type.

5. A computer program product for processing images captured in a retail store and automatically withholding false planogram incompliance notifications, the computer program product embodied in a non-transitory computer-readable medium and including instructions for causing at least one processor to execute a method comprising:

receiving first image data from a plurality of image sensors mounted in the retail store, wherein the first image data depicts a plurality of products displayed on at least one store shelf;

analyzing the first image data using a machine learning model to identify at least one product type associated with the plurality of products displayed on at least one store shelf and to determine a placement of products of the at least one product type on the at least one store shelf;

accessing, in a database, at least one planogram describing a desired placement of products of the at least one product type on shelves of a retail store;

identifying, using a discrepancy detection module, a discrepancy between the determined placement of products associated with the identified at least one product type and the desired placement of products associated with the identified at least one product type determined from the at least one planogram;

receiving second image data captured by the plurality of image sensors prior to capturing the first image data;

analyzing the second image data to identify a past placement of the products of the at least one product type;

determining, using a notice module, whether a notice-override condition exists relative to the identified at least one product type based on the identified discrepancy and the past placement of the products of the at least one product type;

if a notice-override condition exists relative to the identified at least one product type, withholding issuance of a user-notification associated with the identified discrepancy between the determined placement of products and the desired placement of products; and

if a notice-override condition is not determined to exist relative to the identified at least one product type, issuing a user-notification associated with the identified discrepancy between the determined placement of products and the desired placement of products, using a network interface.

6. The computer program product of claim 5 , wherein the second image data corresponds to a last time that the at least one store shelf was organized by a store employee.

7. The computer program product of claim 6 , wherein the determination whether a notice-override condition exists is based on a level of experience of the store employee.

8. The computer program product of claim 5 , wherein the determination whether a notice-override condition exists is further based on inventory information associated with the identified at least one product type.

9. The computer program product of claim 5 , wherein the determination whether a notice-override condition exists is further based on reference information associated with the identified at least one product type from other retail stores.

10. The computer program product of claim 5 , wherein the method further includes analyzing the image data to determine information about products placed adjacent to the products associated with the identified at least one product type, and wherein the determination whether a notice-override condition exists is further based on the information about the products placed adjacent to the products associated with the identified at least one product type.

11. The computer program product of claim 5 , wherein the method further includes:

determining a certainty level for the notice-override condition; and

issuing a user-notification indicating that an event of planogram incompliance associated with the at least one store shelf had occurred when the certainty level is less than a threshold.

12. The computer program product of claim 11 , wherein the determination of the certainty level is further based on the at least one identified product type.

13. The computer program product of claim 11 , wherein the threshold is based on the desired placement of products associated with the at least one store shelf.

14. The computer program product of claim 11 , wherein the method further includes:

receiving additional image data captured at a second time after a determination to withhold issuance of the user-notification based on the existence of the notice-override condition;

analyzing the additional image data to determine an updated certainty level for the notice-override condition; and

issuing a user-notification indicating that an event of planogram incompliance associated with the at least one store shelf had occurred when the updated certainty level is less than a second threshold.

15. The computer program product of claim 5 , wherein the method further comprising:

determining patterns of product availability of product types based on accessed information from one or more databases; and

using the patterns of product availability to determine the existence of the notice-override condition.

16. The computer program product of claim 5 , wherein the method further comprising:

determining patterns of product confusion based on accessed information from one or more databases; and

using the patterns of product confusion to determine the existence of the notice-override condition.

17. The computer program product of claim 16 , wherein the method further includes:

upon issuance of a user-notification associated with the discrepancy receiving data confirming that the discrepancy resulted from erroneous identification of the at least one product type; and

updating the patterns of product confusion to reflect that the discrepancy resulted from erroneous identification of the at least one product type.

18. The computer program product of claim 16 , wherein the method further includes:

receiving additional image data captured at a second time after a determination to withhold issuance of the user-notification due to existence of the notice-override condition;

analyzing the additional image data to determine that the discrepancy associated with the at least one store shelf disappeared without intervention of a store employee; and

updating the patterns of product confusion to reflect that the discrepancy resulted from erroneous identification of the at least one product type.

19. A method for processing images captured in a retail store and automatically identifying situations to withhold planogram incompliance notifications, the method comprising:

receiving first image data from a plurality of image sensors mounted in the retail store, wherein the first image data depicts a plurality of products displayed on at least one store shelf;

analyzing the first image data using a machine learning model to identify at least one product type associated with the plurality of products displayed on at least one store shelf and to determine a placement of products of the at least one product type on the at least one store shelf;

accessing, in a database, at least one planogram describing a desired placement of products of the at least one product type on shelves of a retail store;

identifying, using a discrepancy detection module, a discrepancy between the determined placement of products associated with the identified at least one product type and the desired placement of products associated with the identified at least one product type determined from the at least one planogram;

receiving second image data captured by the plurality of image sensors prior to capturing the first image data;

analyzing the second image data to identify a past placement of the products of the at least one product type;

determining, using a notice module, whether a notice-override condition exists relative to the identified at least one product type due to misidentification of the product based on the identified discrepancy and the past placement of the products of the at least one product type;

if a notice-override condition exists relative to the identified at least one product type, withholding issuance of a user-notification associated with the identified discrepancy between the determined placement of products and the desired placement of products; and

if a notice-override condition is not determined to exist relative to the identified at least one product type, issuing a user-notification associated with the identified discrepancy between the determined placement of products and the desired placement of products.

20. The system of claim 1 , wherein the machine learning model is trained using training image data and labels indicating product types of products depicted in the training image data.

Assignments (3)
RELEASE OF SECURITY INTEREST Recorded Feb 10, 2026
From: COMPUTERSHARE TRUST COMPANY, N.A., AS ADMINISTRATIVE AGENT
To: TRAX TECHNOLOGY SOLUTIONS PTE. LTD.; SHOPKICK, INC.
Reel/Frame 074717/0014 →
SECURITY INTEREST Recorded Sep 22, 2023
From: TRAX TECHNOLOGY SOLUTIONS PTE. LTD.; SHOPKICK, INC.
To: COMPUTERSHARE TRUST COMPANY, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 065016/0744 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 2, 2021
From: ADATO, YAIR; TAIG, RAN
To: TRAX TECHNOLOGY SOLUTIONS PTE LTD.
Reel/Frame 057371/0138 →
Continuity (6)
Continuation PCTUS2020021144 · Mar 5, 2020
Provisional Application 62879565 · Jul 29, 2019
Provisional Application 62872751 · Jul 11, 2019
Provisional Application 62829160 · Apr 4, 2019
Provisional Application 62814339 · Mar 6, 2019
Related Publication 20210398199A1 · Dec 23, 2021