Tracking shoppers and employees
View Patent ↗The system and method discussed herein can capture images from one or more video streams of a store area, can use deep learning to identify people in the images as being a store employee or a shopper, and can use the deep learning to track movement of the people within the store. The tracked movement can provide information that is useful to operators of the store, such as where store employees are, how long they have been in certain areas of the store, which areas of the store need more employees, where most shoppers are concentrated within the store, which areas of the store are popular, and so forth. The system and method can provide instructions to employees on mobile devices or kiosks, in response to the employee locations and activity in the store area. The system and method can also log the movement information, for downstream use.
1. A system, comprising:
a video camera positioned to capture a video stream of a first confined area and a second confined area, wherein the first confined area is at least one of an entrance or an exit of a store;
a video interface configured to receive the video stream; and
a processor coupled to the video interface and configured to execute computing instructions to perform data processing activities, the data processing activities comprising:
receiving a series of images from the video stream;
identifying a first person in the first confined area, wherein the first person is a shopper, and wherein the first confined area is an entrance to the store;
tracking from the series of images, a movement of the first person throughout the store;
determining, from the series of images, a location of the first person in a second confined area;
determining, from the series of images, a location of a second person in the second confined area, wherein the second person is an employee of the store;
determining, from the location of the first person and the location of the second person in the second confined area, an instruction for the second person;
directing the instruction to the second person; and
determining, from the series of images, when the first person exits the store though the first confined area.
2. The system of claim 1 , wherein the processor is configured to use a first convolutional neural network to determine the location of the first person and the second person in the second confined area.
3. The system of claim 1 , wherein the data processing activities further comprise:
log at least one of the first person or the second person as an entry in a database, such that the entry is created when at least one of the first person or the second person is first detected within at least one of the first confined area or the second confined area, and the entry is deleted when the at least one of the first person or the second person is determined to be absent from the at least one of the first confined area or the second confined area.
4. The system of claim 3 , wherein the data processing activities further comprise:
create entries only for one or more specified entry areas within the at least one of the first confined area or the second confined area; and
delete entries only for one or more specified exit areas within the at least one of the first confined area or the second confined area.
5. The system of claim 1 ,
further comprising a second video camera positioned to capture a second video stream of at least a portion of the confined area;
wherein the video interface is further configured to receive the second video stream; and
wherein the data processing activities further comprise:
receiving a second series of images from the second video stream; and
determining, from the series of images and the second series of images, the location of at least one of the first person or the second person in the second confined area.
6. The system of claim 5 , wherein the processor does not rely on spatial coordinates of the plurality of video cameras to determine the locations of people in the confined area.
7. The system of claim 1 , second confined area is a shopping area of a store.
8. The system of claim 7 , wherein the data processing activities further comprise:
automatically determining, from the series of images, whether at least one of the first person or the second person in the confined area is the shopper or the employee of the store, based on clothing worn by the at least one of the first person or the second person.
9. The system of claim 1 , wherein directing the instruction includes sending a message to a smart phone of the second person instructing the second person to attend to an area of the store that requires more store clerks.
10. The system of claim 8 , wherein the processor is configured to use a second convolutional neural network to analyze the clothing worn by the at least one of the first person or the second person to automatically determine if the at least one of the first person or the second person is a shopper or an employee of the store.
11. The system of claim 10 , wherein the second convolutional neural network computes a graham matrix for the at least one of the first person or the second person to determine whether the at least one of the first person or the second person is a shopper or an employee of the store.
12. The system of claim 11 , wherein the graham matrix includes a two-dimensional vector of confidence values, the two-dimensional vector having a supremum, the supremum having an index in the two-dimensional vector, the index indicating whether the at least one of the first person or the second person is a shopper or an employee of the store.
13. A method, comprising:
receiving a series of images from at least one video stream of a first confined area and a second confined area of a store, wherein the first confined area is at least one of an entrance or an exit of the store; and
for each image of the at least one video stream:
using a first convolutional neural network to perform single shot objection detection for people and return all the people present in the image; and
for each person present in the image, using a second convolutional neural network to analyze clothing won by the person to automatically determine if the person is a shopper or an employee of the store;
analyzing the received images to track motion of the shoppers within the second confined area;
analyzing the received images to track motion of the employees of the store within the second confined area;
determining from the tracked motion that a location within the second confined area requires more store clerks;
determining that a first employee is in the second confined area but not in the location within the second confined area;
automatically sending a message to the first employee's smart phone instructing the first employee to attend to the location within the second confined area; and
determining when the shoppers exit the store by passing through the first confined area.
14. The method of claim 13 , further comprising logging people as entries in a database, such that an entry is created when a person is first detected within at least one of the first confined area or the second confined area the confined area, and the entry is deleted when the person is determined to be absent from the at least one of the first confined area or the second confined area the confined area.
15. The method of claim 13 , further comprising logging people as entries in a database, such that an entry is created when a person is first detected within one or more specified entry areas within at least one of the first confined area or the second confined area, and the entry is deleted when the person is last detected to be within one or more specified exit areas within at least one of the first confined area or the second the confined are.
16. A system, comprising:
a video camera positioned to capture a video stream of a first confined area and a second confined area;
a video interface configured to receive the video stream; and
a processor coupled to the video interface and configured to execute computing instructions to perform data processing activities, the data processing activities comprising:
receiving a series of images from at least one video stream of a confined area of a store;
identifying a first person in the first confined area, wherein the first person is a shopper, and wherein the first confined area is at least one of an entrance or an exit to the store;
tracking, using the series of images, a movement of the first person throughout the store;
determining, from the series of images, a location of the first person in a second confined area;
determining, from the series of images, a location of a second person in the second confined area, wherein the second person is an employee of the store;
automatically determining, from the series of images, whether at least one of the first person or the second person in the second confined area is the shopper or the employee of the store, based on clothing worn by the at least one of the first person or the second person;
determining, from the location of the first person and the location of the second person in the second confined area, that an area of the store the location of the first person in the second confined area requires more store clerks;
sending a message to a smart phone of the second person instructing the second person to attend to the location of the first person within the second confined area; and
determining, from the series of images, when the first person exits the store through the first confined area.
17. The system of claim 16 ,
further comprising a second video camera positioned to capture a second video stream of at least a portion of at least one of the first confined area or the second confined area;
wherein the video interface is further configured to receive the second video stream; and
wherein the data processing activities further comprise:
receiving a second series of images from the second video stream; and
determining, from the series of images and the second series of images, the locations of the at least one of the first person or the second person in the at least one of the first confined area or the second confined area people in the confined area.
18. The system of claim 16 ,
wherein the at least one video stream includes a plurality of video streams; and
further comprising a plurality of video cameras, coupled to the processor, and positioned to capture the plurality of video streams of the confined area, wherein the processor does not rely on spatial coordinates of the plurality of video cameras to determine the locations of people in the confined area.
19. The system of claim 16 , wherein the data processing activities further comprise:
logging people as entries in a database, such that an entry is created when at least one of the first person or the second person is first detected within at least one of the first confined area or the second confined area, and the entry is deleted when the at least one of the first person or the second person is determined to be absent from the at least one of the first confined area or the second confined area.
20. The system of claim 19 , wherein the data processing activities further comprise:
creating entries only for one or more specified entry areas within the at least one of the first confined area or the second confined area; and
deleting entries only for one or more specified exit areas within the at least one of the first confined area or the second confined area.