Shopping basket monitoring using computer vision
A system for monitoring shopping carts uses cameras to generate images of the carts moving in a store. In some implementations, cameras may additionally or alternatively be mounted to the shopping carts and configured to image cart contents. The system may use the collected image data, and/or other types of sensor data (such as the store location at which an item was added to the basket), to classify items detected in the shopping carts. For example, a trained machine learning model may classify item in a cart as “non-merchandise,” “high theft risk merchandise,” “electronics merchandise,” etc. When a shopping cart approaches a store exit without any indication of an associated payment transaction, the system may use the associated item classification data, optionally in combination with other data such as cart path data, to determine whether to execute an anti-theft action, such as locking a cart wheel or activating a store alarm. The system may also compare the classifications of cart contents to payment transaction records (or summaries thereof) to, e.g., detect underpayment events.
1 . A system for reducing push-out theft, comprising:
a camera positioned to capture images of contents of a shopping cart associated with a store, the camera mounted in a fixed location in the store;
a wheel assembly attached to the shopping cart, wherein the wheel assembly senses, and reports on a wireless network, wheel rotation events; and
a computing system comprising one or more processors, the computing system programmed to implement at least:
an item classifier that assigns classifications to items detected in the images using a trained machine learning model;
a correlator that uses the wheel rotation events to match an identifier of the shopping cart to a cart track detected in a sequence of images captured by the camera; and
an exit manager that determines whether to authorize the shopping cart to exit the store based on at least (1) a classification assigned by the item classifier to at least one item in the shopping cart, and (2) indicia of whether the at least one item has been paid for.
2 . The system of claim 1 , wherein the item classifier is configured to classify items as merchandise versus non-merchandise items.
3 . The system of claim 1 , wherein the item classifier is configured to assign merchandise category classifications to items.
4 . The system of claim 1 , wherein the item classifier is configured to classify an item added to the shopping cart based at least partly on a store location of the shopping cart at a time the item is added.
5 . The system of claim 1 , wherein the computing system is further programmed to:
identify, based on a monitored location of the shopping cart, a payment point potentially used to pay for merchandise in the shopping cart; and
compare a record of a transaction executed by the pay point to item classifications assigned by the item classifier to items detected in the shopping cart.
6 . The system of claim 5 , wherein the computing system is further configured to detect, based on the comparison, an underpayment event in which a customer pays for less than all items in the shopping cart.
7 . The system of claim 5 , wherein the computing system is further configured to determine whether the shopping cart returns to a merchandise area after payment without first exiting the store.
8 . The system of claim 1 , wherein the camera has a wavelength response that encompasses near IR and/or near UV.
9 . A process for reducing push-out theft, comprising:
monitoring, with a camera, a location of a shopping cart in a store;
sensing, with a wheel assembly on the shopping cart, wheel rotation events;
reporting the wheel rotation events on a wireless network;
matching, using the wheel rotation events, an identifier of the shopping cart to a cart track detected in a sequence of images captured by the camera;
detecting an event in which an item is added to the shopping cart;
assigning a classification to the item based on at least one of the following: (1) a location of the shopping cart in the store at a time of the event, as determined from said monitoring, and (2) an image captured by a camera of the item in the shopping cart;
subsequently, detecting that the shopping cart is exiting the store without indicia of an associated payment transaction; and
in response to determining that the shopping cart is exiting, determining, based at least partly on the classification assigned to the item, whether to perform an anti-theft action;
said process performed under control of program instructions executed by one or more processors.
10 . The process of claim 9 , wherein detecting said event comprises detecting, with a vibration sensor mounted to the shopping cart, vibration caused by the item being added to the shopping cart.
11 . The process of claim 9 , wherein the shopping cart comprises a camera module having a camera and a processor, and wherein detecting said event comprises, by the camera module, comparing a first image captured by the camera to a second, subsequent image captured by the camera to determine whether contents of the shopping cart have changed.
12 . The process of claim 11 , wherein the camera module further comprises a wireless transceiver, and the process further comprises, by the camera module, uploading the second image, or a representation of the second image, on a wireless network for analysis in response to detecting a change to basket contents.
13 . The process of claim 9 , wherein assigning a classification to the item comprises using a trained machine learning model to classify the item as merchandise versus non-merchandise.
14 . The process of claim 9 , wherein assigning a classification to the item comprises determining a merchandise identifier of the item.
15 . The process of claim 9 , wherein assigning a classification to the item comprises assigning a merchandise category to the item.
16 . A system for reducing push-out theft in a store, comprising:
a camera positioned to capture images of a shopping cart and the contents of a basket of the shopping cart, the camera mounted in a fixed location in the store;
a wheel assembly attached to the shopping cart, wherein the wheel assembly senses, and reports on a wireless network, wheel rotation events; and
a computing system comprising one or more processors, the computing system programmed to implement at least:
an item classifier that assigns classifications to items detected in the images using a trained machine learning model;
a correlator that uses the wheel rotation events to match an identifier of the shopping cart to a cart track detected in a sequence of the images captured by the camera; and
an exit manager that determines whether to authorize the shopping cart to exit the store based on at least (1) a classification assigned by the item classifier to at least one item in the shopping cart, and (2) the cart track of the shopping cart.
17 . The system of claim 16 , wherein the determination of whether to authorize the shopping cart to exit the store is further based on at least whether the cart track of the shopping cart passed though a pay point.
18 . The system of claim 16 , wherein the determination of whether to authorize the shopping cart to exit the store is further based on at least whether the cart track of the shopping cart passed though region of store classified as a high value area.
19 . The system of claim 16 , wherein the item classifier is configured to classify items as merchandise versus non-merchandise items.
20 . The system of claim 16 , wherein the item classifier is configured to assign merchandise category classifications to items.
21 . The system of claim 16 , wherein the item classifier is configured to classify an item added to the shopping cart based at least partly on a store location of the shopping cart at a time the item is added.
22 . The system of claim 16 , wherein the computing system is further programmed to:
identify, based on a monitored location of the shopping cart, a payment point potentially used to pay for merchandise in the shopping cart; and
compare a record of a transaction executed by the pay point to item classifications assigned by the item classifier to items detected in the shopping cart.