IP Library Granted Patent US 10,943,128
Granted Patent B2
US 10,943,128 · App. 16/177,744 · Granted Mar 9, 2021

Constructing shopper carts using video surveillance

Inventors: Brent Vance Zucker (Roswell, GA); Stefan Bjelcevic (Roswell, GA); Adam Justin Lieberman (Suwanee, GA)
Assignee: NCR Corporation
G06K9/00771G06K7/10366G06K7/1413G06K7/1417G06K9/00362G06Q20/12G06Q30/0635G06T7/20H04N5/247G06K9/6201G06T2207/10016G06T2207/30196G06T2207/30232
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,943,128
App. No.
16/177,744
Granted
Mar 9, 2021
Kind
B2
Abstract

A system can visually track which items in a store are selected for purchase by a shopper. The system can form a virtual shopping cart by analyzing multiple images, over time, to determine which purchasable items are located with the shopper, such as in a physical shopping cart, in a basket, or held by the shopper. By analyzing multiple images, over time, the system can account for items misidentified in one or more images, or fully or partially obscured in one or more images as the shopper traverses the store. Alternatively, the system can form a virtual shopping cart by analyzing instances in which a purchasable item is removed from a shelf or placed on a shelf. Items removed from, but not returned to, a shelf can be considered to be selected for purchase. The system can include a frictionless checkout that charges the shopper for the selected items.

Claims (37)

1. A method, comprising:

capturing, with video surveillance of a store, an image of a shopper;

determining, using a processor coupled to a video interface, that the shopper is not a person already present in the store;

associating, using the processor, a virtual shopping cart with the shopper;

automatically tracking, the video surveillance of a store, movement of the shopper throughout the store and items in the store selected for purchase by the shopper, wherein the tracking includes determining a historical position of the selected items to determine a velocity of the selected items, wherein, when the velocity of the selected items is in a direction away from a shelf, the tracking includes storing a data representation of the selected items in the virtual shopping cart, and wherein when the velocity of the selected items is in a direction toward the shelf, the tracking includes removing the selected items from the virtual shopping cart; and

charging, through a frictionless check-out of the store, an account associated with the shopper for purchase of the selected items, wherein the frictionless check-out includes reconciling the data representation of the selected items in the virtual shopping cart.

2. The method of claim 1 , wherein automatically tracking the items in the store selected for purchase by the shopper comprises:

capturing images from at least one video stream of a shopping area of the store;

automatically recognizing the shopper in a plurality of the captured images;

visually identifying, from each of the plurality of captured images, purchasable items located within a bounding area around the shopper; and

using a sliding time window for the plurality of captured images to correct purchasable items that are misidentified or obscured in one or more of the plurality of captured images.

3. The method of claim 2 , wherein using the sliding time window comprises:

automatically analyzing a plurality of historical images, the historical images being a specified number of images from the video stream captured immediately prior to a current captured image; and

visually identifying an inventory of any purchasable items located within a bounding area around the shopper, for each historical image.

4. The method of claim 3 , wherein using the sliding time window further comprises:

identifying a misidentified inventory of a first historical image of the plurality of historical images, wherein an inventory of a historical image taken before the first historical image matches an inventory of a historical image taken after the first historical image but differs from the misidentified inventory of the first historical image.

5. The method of claim 4 , wherein using the sliding time window further comprises:

ignoring the misidentified inventory, such that the items selected for purchase match the inventories of the historical images taken before and after the first historical image.

6. The method of claim 5 , wherein the video surveillance of the store comprises:

with at least one overhead camera directed generally downward, capturing at least one video stream of a shopping area of the store.

7. The method of claim 6 , wherein each overhead camera has a field of view that is centered within twenty degrees of a vertical axis.

8. The method of claim 7 , wherein each overhead camera has a field of view that is centered within ten degrees of the vertical axis.

9. A system, comprising:

at least one video camera positioned to capture at least one video stream of a store;

a video interface configured to receive the at least one video stream; and

a processor coupled to the video interface;

memory, including instructions stored thereon, which, when executed by the processor, cause the processor to perform data processing activities, the data processing activities comprising:

capture an image of a shopper;

determine that the shopper is not a person already present in the store;

associating a virtual shopping cart with the shopper;

automatically tracking, from the at least one video stream, movement of the shopper throughout the store and items in the store selected for purchase by the shopper, wherein the tracking includes determining a historical position of the selected items to determine a velocity of the selected items, wherein, when the velocity of the selected items is in a direction awa from a shelf, the tracking includes storing a data representation of the selected items in the virtual shopping cart, and wherein when the velocity of the selected items is in a direction toward the shelf, the tracking includes removing the selected items from the virtual shopping cart; and

charging, through a frictionless check-out of the store, an account associated with the shopper for purchase of the selected items, wherein the frictionless check-out includes reconciling the data representation of the selected items in the virtual shopping cart.

10. The system of claim 9 , wherein:

each video camera has a field of view centered around a downward-facing generally vertical axis; and

automatically tracking the items in the store comprises:

visually identifying an inventory of any purchasable items located within a bounding area around the shopper, for multiple images from the video stream; and

relying on a sliding window of images from the video stream to correct a misidentified inventory of an image from the video stream.

Assignments (6)
CHANGE OF NAME Recorded Dec 7, 2023
From: NCR CORPORATION
To: NCR VOYIX CORPORATION
Reel/Frame 065820/0704 →
RELEASE OF PATENT SECURITY INTEREST Recorded Oct 25, 2023
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: NCR VOYIX CORPORATION
Reel/Frame 065346/0531 →
SECURITY INTEREST Recorded Oct 25, 2023
From: NCR VOYIX CORPORATION
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 065346/0168 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PROPERTY NUMBERS SECTION TO REMOVE PATENT APPLICATION: 15000000 PREVIOUSLY RECORDED AT REEL: 050874 FRAME: 0063. ASSIGNOR(S) HEREBY CONFIRMS THE SECURITY INTEREST. Recorded Apr 12, 2021
From: NCR CORPORATION
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 057047/0161 →
SECURITY INTEREST Recorded Oct 29, 2019
From: NCR CORPORATION
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 050874/0063 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2018
From: ZUCKER, BRENT VANCE; BJELCEVIC, STEFAN; LIEBERMAN, ADAM JUSTIN
To: NCR CORPORATION
Reel/Frame 047383/0745 →