IP Library Granted Patent US 10,671,834
Granted Patent B2
US 10,671,834 · App. 16/354,601 · Granted Jun 2, 2020

Using context to update product models

Inventors: Yair Adato (Kfar Shmuel, IL); Aviv Eisenschtat (Ramat Hasharon, IL); Dolev Pomeranz (Hod Hasharon, IL); Ziv Mhabary (Tel Aviv, IL); Daniel Shimon Cohen (Hoboken, NJ); Osnat Yanushevsky (Ramat Hasharon, IL)
Assignee: TRAX TECHNOLOGY SOLUTIONS PTE LTD.
G06K9/00201G06F16/23G06F16/235G06F16/288G06F16/55G06F16/583G06F16/5846G06F16/90335G06F17/18G06K9/00362G06K9/00624G06K9/00671G06K9/00771G06K9/325G06K9/6231G06Q10/06311G06Q10/06316G06Q10/063112G06Q10/087G06Q10/0875G06Q20/203G06Q30/0629G06Q30/0639G06Q30/0643G06Q50/28G06T7/0002G06T7/13G06T7/20G06T7/521G06T7/55G06T7/70G06T7/75G08B21/18G08B21/182H04N5/2252H04N5/2253H04N5/23203H04N5/23219H04N5/23229H04N5/247G06K2209/01G06K2209/17G06T2207/30196G06T2207/30232G06T2207/30242
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,671,834
App. No.
16/354,601
Granted
Jun 2, 2020
Kind
B2
Abstract

A system for processing images captured in a retail store is provided. The system may include a processor configured to: access a database storing a group of product models; receive an image depicting at least part of a store shelf having a plurality of products of a same type displayed thereon; analyze the image and determine a first candidate type of the products based on the group of product models and the image analysis; determine a first confidence level associated with the first candidate type; when the first confidence level is below a confidence threshold, determine a second candidate type of the products using contextual information; determine a second confidence level associated with the determined second candidate type of the plurality of products; and when the second confidence level is above the confidence threshold, initiate an action to update the group of product models stored in the database.

Claims (47)

1. A system for processing images captured in a retail store, the system comprising:

at least one processor configured to:

access a database storing a group of product models, each relating to at least one product in the retail store;

receive at least one image depicting at least part of at least one store shelf having a plurality of products of a same type displayed thereon;

analyze the received at least one image and determine a first candidate type of the plurality of products based on the group of product models and the image analysis;

determine a first confidence level associated with the determined first candidate type of the plurality of products;

when the first confidence level associated with the first candidate type is below a confidence threshold, determine a second candidate type of the plurality of products using contextual information;

determine a second confidence level associated with the determined second candidate type of the plurality of products; and

when the second confidence level associated with the second candidate type is above the confidence threshold, initiate an action to update the group of product models stored in the database based on the contextual information, wherein updating the group of product models includes deleting or deactivating at least one product models of the group of product models.

2. The system of claim 1 , wherein when the first confidence level of the first candidate type is below the confidence threshold, the at least one processor is further configured to:

provide a visual representation of the plurality of products to a user;

select multiple alternative product types based on confidence levels corresponding to the multiple alternative product types;

provide the selected multiple alternative product types for the user to select from;

receive input from the user indicating a type of the plurality of products in a form of a selection of an option from the multiple alternative product types; and

determine the second candidate type of the plurality of products using the received input.

3. The system of claim 1 , wherein the at least one processor is further configured to select the confidence threshold based on a combination of a location of the plurality of products in the retail store and a day of year.

4. The system of claim 1 , wherein the at least one processor is further configured to determine the first confidence level based on a location of the at least one store shelf in the retail store.

5. The system of claim 1 , wherein the contextual information used to determine the second candidate type includes a brand name of the plurality of products.

6. The system of claim 1 , wherein the contextual information used to determine the second candidate type includes a price associated with the plurality of products.

7. The system of claim 1 , wherein the contextual information used to determine the second candidate type includes at least one logo appearing on the product.

8. The system of claim 1 , wherein the contextual information used to determine the second candidate type includes a probability for a first product type and a second product type to be placed in proximity to each other determined by analyzing images from multiple stores.

9. The system of claim 1 , wherein the contextual information used to determine the second candidate type includes information from a catalog of the retail store.

10. The system of claim 1 , wherein the at least one processor is further configured to determine the first confidence level based on types of products adjacent the plurality of products.

11. The system of claim 1 , wherein the first candidate type and the second candidate type are associated with the same type of product that changed an appearance attribute, and the contextual information used to determine the second candidate type includes an indication that the plurality of products are associated with a new appearance attribute.

12. The system of claim 1 , wherein the at least one processor is further configured to select the action to initiate, from among a plurality of available actions, based on the determined confidence level of the second candidate type.

13. The system of claim 1 , wherein the action to update the group of product models includes adding a new product model to the group of product models, the new product model being representative of a previously unidentified type of products and found in a repository of product models from other retail stores.

14. The system of claim 1 , wherein the action to update the group of product models includes replacing an existing product model from the group of product models with at least one new product model, the existing product model and the new product model being associated with a same product type, and wherein the at least one processor is further configured to, after replacing the existing product model, reactivate the existing product model in response to an indication from a calendar.

15. The system of claim 1 , wherein the action to update the group of product models includes modifying an existing product model from a group of product models, the existing product model comprises a parameter of an image convolution function, the modification to the existing product model being based on a detected a change in an appearance attribute of the plurality of products and includes a change to the parameter of the image convolution function.

16. The system of claim 1 , wherein the action to update the group of product models includes deactivating an existing product model from group of product models, a deactivation of the existing product model being based on a detected a change in an appearance attribute of the plurality of products.

17. A method for processing images captured in a retail store, the method comprising:

accessing a database storing a group of product models, each relating to at least one product in the retail store;

receiving at least one image depicting at least part of at least one store shelf having a plurality of products of a same type displayed thereon;

analyzing the at least one image and determining a first candidate type of the plurality of products based on the group of product models and the image analysis;

determining a first confidence level associated with the determined first candidate type of the plurality of products;

when the first confidence level associated with the first candidate type is below a confidence threshold, determining a second candidate type of the plurality of products using contextual information;

determining a second confidence level associated with the determined second candidate type of the plurality of products; and

when the second confidence level associated with the second candidate type is above the confidence threshold, initiating an action to update the group of product models stored in the database based on the contextual information, wherein updating the group of product models includes deleting or deactivating at least one product models of the group of product models.

18. The method of claim 17 , further comprising selecting the confidence threshold based on a portion of the retail store, wherein a higher confidence threshold is selected for more profitable portions of the retail store.

19. The method of claim 17 , further comprising selecting the confidence threshold based on a time, wherein a higher confidence threshold is selected during a stocktaking period.

20. A computer program product for processing images captured in a retail store embodied in a non-transitory computer-readable medium and executable by at least one processor, the computer program product including instructions for causing the at least one processor to execute a method comprising:

accessing a database storing a group of product models, each relating to at least one product in the retail store;

receiving at least one image depicting at least part of at least one store shelf having a plurality of products of a same type displayed thereon;

analyzing the at least one image and determining a first candidate type of the plurality of products based on the group of product models and the image analysis;

determining a first confidence level associated with the determined first candidate type of the plurality of products;

when the first confidence level associated with the first candidate type is below a confidence threshold, determining a second candidate type of the plurality of products using contextual information;

determining a second confidence level associated with the determined second candidate type of the plurality of products; and

when the second confidence level associated with the second candidate type is above the confidence threshold, initiating an action to update the group of product models stored in the database based on the contextual information, wherein updating the group of product models includes deleting or deactivating at least one product models of the group of product models.

Assignments (7)
RELEASE OF SECURITY INTEREST Recorded Feb 10, 2026
From: COMPUTERSHARE TRUST COMPANY, N.A., AS ADMINISTRATIVE AGENT
To: TRAX TECHNOLOGY SOLUTIONS PTE. LTD.; SHOPKICK, INC.
Reel/Frame 074717/0014 →
RELEASE OF SECURITY INTEREST Recorded Sep 26, 2023
From: ALTER DOMUS (US) LLC
To: TRAX TECHNOLOGY SOLUTIONS PTE. LTD.; SHOPKICK INC.
Reel/Frame 065028/0871 →
SECURITY INTEREST Recorded Sep 22, 2023
From: TRAX TECHNOLOGY SOLUTIONS PTE. LTD.; SHOPKICK, INC.
To: COMPUTERSHARE TRUST COMPANY, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 065016/0744 →
RELEASE OF SECURITY INTEREST AT REEL/FRAME NO. 054048/0646 Recorded Oct 27, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: TRAX TECHNOLOGY SOLUTIONS PTE. LTD; SHOPKICK, INC.; CVDM SOLUTIONS SAS; TRAX RETAIL, INC.
Reel/Frame 057944/0338 →
SECURITY INTEREST Recorded Oct 15, 2021
From: TRAX TECHNOLOGY PTE. LTD.; SHOPKICK, INC.
To: ALTER DOMUS (US) LLC
Reel/Frame 058184/0438 →
SECURITY INTEREST Recorded Oct 14, 2020
From: TRAX TECHNOLOGY SOLUTIONS PTE. LTD.; SHOPKICK, INC.; CVDM SOLUTIONS SAS; TRAX RETAIL, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054048/0646 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 15, 2019
From: ADATO, YAIR; EISENSCHTAT, AVIV; POMERANZ, DOLEV; MHABARY, ZIV; COHEN, DANIEL SHIMON; YANUSHEVSKY, OSNAT
To: TRAX TECHNOLOGY SOLUTIONS PTE LTD.
Reel/Frame 048609/0962 →
Continuity (5)
Continuation PCTUS2019013054 · Jan 10, 2019
Provisional Application 62615512 · Jan 10, 2018
Provisional Application 62681718 · Jun 7, 2018
Provisional Application 62695469 · Jul 9, 2018
Related Publication 20190213212A1 · Jul 11, 2019
Cited By (6)
US 12,200,342 US 12,405,700 US 12,542,862 US 12,602,154 US 12,619,303 US 12,646,347