IP Library Granted Patent US 11,004,113
Granted Patent B1
US 11,004,113 · App. 16/994,440 · Granted May 11, 2021

Electronic device with cameras and display for in-store messaging

Inventor: Herb Sorensen (Corbett, OR)
Assignee: SHOPPER SCIENTIST LLC
G06Q30/0261G06F1/1605G06F3/013G06F3/147G06N3/08G09F9/30G09F27/005G09F2027/001
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,004,113
App. No.
16/994,440
Granted
May 11, 2021
Kind
B1
Abstract

An electronic device for a store is provided comprising a front-facing camera, a rear-facing camera, a display, and processing circuitry configured to receive an identity of at least one target product that meets one or more predetermined product criteria. The processing circuitry is further configured to capture a first image via the rear-facing camera and determine that the target product proximate the electronic device is depicted in the first image, and to capture a second image via the front-facing camera and detect a shopper in a vicinity of the electronic device is depicted in the second image. The processing circuitry is further configured to in response to determining that the target product proximate the electronic device is depicted in the first image and detecting that the shopper in a vicinity of the electronic device is depicted in the second image, displaying a message related to the target product.

Claims (77)

1. An electronic device for a store comprising:

a front-facing camera;

a rear-facing camera;

a display; and

processing circuitry configured to:

receive wirelessly from a remote computing device an identity of at least one target product that meets one or more predetermined product criteria;

capture a first image via the rear-facing camera;

determine that the target product proximate the electronic device is depicted in the first image;

capture a second image via the front-facing camera;

detect a shopper in a vicinity of the electronic device is depicted in the second image; and

in response to determining that the target product proximate the electronic device is depicted in the first image and detecting that the shopper in a vicinity of the electronic device is depicted in the second image, displaying a message related to the target product on the display, wherein

the remote computing device is further configured to store a stored image of the target product, and to store the message related to the target product, and

determining that the target product proximate the electronic device is depicted in the first image includes:

extracting from a series of rear images a plurality of features, and

comparing the first image to the stored image of the target product.

2. The electronic device for a store of claim 1 , further configured to, via the rear-facing camera, determine a number of the target product remaining and notify a store employee if the number is below a predetermined limit.

3. The electronic device for a store of claim 1 , further configured to, via the rear-facing camera, determine a number of the target products remaining and inhibit display of a second message when the number of the target product remaining is below a predetermined limit.

4. The electronic device for a store of claim 1 , wherein the remote computing device is configured to record sales logs of target products in the store.

5. The electronic device for a store of claim 1 , wherein detecting that the shopper is within a vicinity of the electronic device includes detecting that the shopper is within a predetermined distance of the electronic device and that a gaze of the shopper is within a predetermined angle with respect to a plane of the display.

6. The electronic device for a store of claim 1 , wherein the one or more predetermined product criteria include: being one of a plurality of top selling products or being selected by an administrator.

7. The electronic device for a store of claim 1 , further configured to:

record respective positions of one or more shoppers within the store; and

convey the respective positions of the one or more shoppers to the remote computing device.

8. The electronic device for a store of claim 1 , further configured to:

record a duration of the shopper in a vicinity of the electronic device; and

convey the duration of the shopper in a vicinity of the electronic device to the remote computing device.

9. The electronic device for a store of claim 1 , further comprising a speaker and further configured to deliver an auditory message.

10. The electronic device for a store of claim 1 , further configured to:

record a change in the first image via the rear-facing camera;

record a pickup of the target product; and

record a duration between displaying the message and the pickup of the target product.

11. The electronic device for a store of claim 10 , wherein

the first image is a rear image;

the second image is a front image; and

the processing circuitry is further configured to, in a training phase:

capture a series of rear images from the rear-facing camera, each of the series of rear images having an associated timestamp;

capture a series of front images from the front-facing camera, each of the series of front images having an associated timestamp; and

generate a training data set with rear images, front images, a first message template, and a second message template;

alternately display the message based on the first message template and the second message template; and

train an artificial intelligence model to predict whether the first message template or the second message template results in a higher pickup.

12. The electronic device for a store of claim 11 , wherein the processing circuitry is further configured to:

in a runtime phase:

display a message related to the target product on the display by outputting, in response to receiving a runtime input, an action in the form of the message according to the message template.

13. The electronic device for a store of claim 10 , wherein:

the first image is a rear image;

the second image is a front image; and

the processing circuitry is further configured to, in a runtime phase:

capture a series of rear images from the rear-facing camera, each of the series of rear images having an associated timestamp;

capture a series of front images from the front-facing camera, each of the series of front images having an associated timestamp;

determine that the target product proximate the electronic device is depicted in the first image by extracting from the series of rear images a plurality of features as input into an artificial intelligence model;

detect that the shopper in a vicinity of the electronic device is depicted in the second image by extracting from the series of front images a plurality of features as input into the artificial intelligence model; and

display the message related to the target product on the display by outputting in response to receiving a runtime input an action in the form of the message according to a message template.

14. The electronic device for a store of claim 13 , wherein the artificial intelligence model has been trained on the plurality of features from the front images as training time input and from the rear images as ground truth classification of the front-facing images at training time.

15. The electronic device for a store of claim 13 , wherein the artificial intelligence model is a reinforcement learning model and uses as a reward a pickup ratio.

16. The electronic device for a store of claim 15 , wherein the pickup ratio is defined as a number of pickups of the target product divided by a number of messages displayed.

17. The electronic device for a store of claim 13 , wherein the message template is optimized by the artificial intelligence model and includes at least one of the following:

a message duration;

a message selection, wherein one of a plurality of predetermined messages is selected by the artificial intelligence model.

18. The electronic device for a store of claim 1 , wherein the message includes a pulsing illuminated textual message or image.

19. An electronic device for a store comprising:

a front-facing camera;

a rear-facing camera;

a display; and

processing circuitry configured to:

receive wirelessly from a remote computing device an identity of at least one target product that meets one or more predetermined product criteria;

capture a first image via the rear-facing camera;

determine that the target product proximate the electronic device is depicted in the first image;

capture a second image via the front-facing camera;

detect a shopper in a vicinity of the electronic device is depicted in the second image; and

in response to determining that the target product proximate the electronic device is depicted in the first image and detecting that the shopper in a vicinity of the electronic device is depicted in the second image, displaying a message related to the target product on the display wherein

displaying a message related to the target product is further conditioned upon one or more additional message conditions determined by at least one of the following:

a shopper density feature;

a product density feature;

a time density feature;

a shopper distance feature; and

a shopper gaze feature; wherein

displaying the message related to the target product on the display is performed in response to determining that the target product proximate the electronic device is depicted in the first image and detecting that the shopper in a vicinity of the electronic device is depicted in the second image, and further in response to determining that the additional message condition is satisfied.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 14, 2020
From: SORENSEN, HERB
To: SHOPPER SCIENTIST LLC
Reel/Frame 053505/0042 →
Cited By (1)
US 12,658,014