Method for augmenting transaction data with visually extracted demographics of people using computer vision
View Patent ↗The present invention is a system and framework for augmenting any retail transaction system with information about the involved customers. This invention provides a method to combine the transaction data records and a customer or a group of customers with the automatically extracted demographic features (e.g., gender, age, and ethnicity), shopping group information, and behavioral information using computer vision algorithms. First, the system detects faces from face view, tracks them individually, and estimates poses of each of the tracked faces to normalize. These facial images are processed by the demographics classification module to determine and record the demographics feature vector. The system detects and tracks customers to analyze the dynamic behavior of the tracked customers so that their shopping group membership and checkout behavior can be recognized. Then the instances of faces and the instances of bodies can be matched and combined. Finally, the transaction data from the transaction data and the demographics, group, and checkout behavior data that belong to the same person or the same group of people are combined.
1. A method for combining automatically detected demographic, group, and behavior features of people in a retail transaction area with transaction data, comprising the following steps of:
a) acquiring facial images of the people from first input images captured by at least a first means for capturing images, including a visual sensing device,
b) determining demographic categories of the facial images to generate a demographic data, using at least a demographic feature extractor,
c) acquiring person images from second input images captured by at least a second means for capturing images, including a visual sensing device, near the transaction area,
d) determining shopping group membership of the people to generate group data from the second input images,
e) analyzing the movement of the people for behavior analysis by determining checkout behaviors of the people based on analysis of the second input images by recognizing interactions of the people with merchandise in the transaction area,
f) associating transaction data with the demographics data, the behavior analysis, and the group data, and
wherein the analysis of the second input images comprises body orientation estimation of the people, proximity calculation between the people and checkout shelves, and foreground object analysis, in the transaction area,
wherein the first means for capturing images comprises a face view camera that captures the facial images of the people waiting in the checkout queue for face detection,
wherein the second means for capturing images comprises a top-down view camera that captures the top-down view and person images of the checkout queue for person detection,
wherein the face view camera and the top-down view camera are placed and oriented so that the image positions of the facial images and the person images are translated into a common world-coordinate system, and
wherein the steps are performed in a control and processing system that is connected to the means for capturing images.
2. The method according to claim 1 ,
wherein the method further comprises a step of detecting and tracking the person by using at least a learning machine to find person images,
wherein the learning machine is trained to find human bodies in a motion foreground region in the second input images.
3. The method according to claim 1 ,
wherein the method further comprises a step of automatically identifying shopping group information of the people based on their movements,
wherein body images of the people are tracked, and
wherein the tracked body images are analyzed to determine whether one or more of the tracked people belong to the shopping group in the top-down view.
4. The method according to claim 1 ,
wherein the method further comprises a step of determining a leader of the shopping group by analyzing facial expressions of the people in the transaction area,
wherein the leader is a person who takes the role of interacting with the cashier and makes the payment, and
wherein the facial image analysis to determine the leader includes the estimation of facial pose and facial expression that represents the degree of attention and emotion of the person.
5. The method according to claim 1 , wherein the method further comprises a step of determining emotional responses of the people to the shopping experience by analyzing the facial expressions of the people in the transaction area.
6. The method according to claim 1 , wherein the method further comprises a step of associating the facial images and the person images by comparing image coordinates and timestamps of the facial images and image coordinates and timestamps of the person images.
7. The method according to claim 1 , wherein the method further comprises a step of associating the transaction data with the demographics data, the behavior data, and the group data based on the timestamps recorded from the transaction terminal and the timestamps recorded from the facial image acquisitions and the person image acquisitions.
8. The method according to claim 1 ,
wherein the method further comprises a step of associating the loyalty card data, transaction data, and shopper data,
wherein the shopper data is measured based on automatic video analytics for visual images of shoppers without any interruption, and
wherein the shopper data is calculated using the facial images from the first means for capturing images and the person images from the second means for capturing images in the checkout queue.
9. The method according to claim 1 ,
wherein the method further comprises a step of analyzing the performance difference among different types of checkout environments,
wherein the types of checkout environments comprise a self-checkout and a cashier serviced checkout, and
wherein the performance comprises sales data of products, average checkout time, and average number of people in the waiting queue per a particular demographic group, based on the demographic category determination from the facial images and analysis of the movement of the people from the person images.
10. An apparatus for combining automatically detected demographic, group, and behavior features of people in a retail transaction area with transaction data, comprising:
a) means for acquiring facial images of the people from first input images captured by at least a first means for capturing images, including a visual sensing device,
b) means for determining demographic categories of the facial images to generate a demographic data, wherein the means for determining demographic categories includes at least a demographic feature extractor,
c) means for acquiring person images from second input images captured by at least a second means for capturing images, including a visual sensing device, near the transaction area,
d) means for determining shopping group membership of the people to generate group data from the second input images,
e) means for analyzing the movement of the people for behavior analysis by determining checkout behaviors of the people based on analysis of the second input images by recognizing interactions of the people with merchandise in the transaction area, and
f) means for associating transaction data with the demographics data, the behavior analysis, and the group data,
wherein the analysis of the second input images comprises body orientation estimation of the people, proximity calculation between the people and checkout shelves, and foreground object analysis, in the transaction area,
wherein the first means for capturing images comprises a face view camera that captures the facial images of the people waiting in the checkout queue for face detection,
wherein the second means for capturing images comprises a top-down view camera that captures the top-down view and person images of the checkout queue for person detection,
wherein the face view camera and the top-down view camera are placed and oriented so that the image positions of the facial images and the person images are translated into a common world-coordinate system, and
wherein the apparatus further comprises a control and processing system that is connected to the means for capturing images.
11. The apparatus according to claim 10 ,
wherein the apparatus further comprises means for detecting and tracking the person by using at least a learning machine to find person images,
wherein the learning machine is trained to find human bodies in a motion foreground region in the second input images.
12. The apparatus according to claim 10 ,
wherein the apparatus further comprises means for automatically identifying shopping group information of the people based on their movements,
wherein body images of the people are tracked, and
wherein the tracked body images are analyzed to determine whether one or more of the tracked people belong to the shopping group in the top-down view.
13. The apparatus according to claim 10 ,
wherein the apparatus further comprises means for determining a leader of the shopping group by analyzing facial expressions of the people in the transaction area,
wherein the leader is a person who takes the role of interacting with the cashier and makes the payment, and
wherein the facial image analysis to determine the leader includes the estimation of facial pose and facial expression that represents the degree of attention and emotion of the person.
14. The apparatus according to claim 10 , wherein the apparatus further comprises means for determining emotional responses of the people to the shopping experience by analyzing the facial expressions of the people in the transaction area.
15. The apparatus according to claim 10 , wherein the apparatus further comprises means for associating the facial images and the person images by comparing image coordinates and timestamps of the facial images and image coordinates and timestamps of the person images.
16. The apparatus according to claim 10 , wherein the apparatus further comprises means for associating the transaction data with the demographics data, the behavior data, and the group data based on the timestamps recorded from the transaction terminal and the timestamps recorded from the facial image acquisitions and the person image acquisitions.
17. The apparatus according to claim 10 ,
wherein the apparatus further comprises means for associating the loyalty card data, transaction data, and shopper data,
wherein the shopper data is measured based on automatic video analytics for visual images of shoppers without any interruption, and
wherein the shopper data is calculated using the facial images from the first means for capturing images and the person images from the second means for capturing images in the checkout queue.
18. The apparatus according to claim 10 ,
wherein the apparatus further comprises means for analyzing the performance difference among different types of checkout environments,
wherein the types of checkout environments comprise a self-checkout and a cashier serviced checkout, and
wherein the performance comprises sales data of products, average checkout time, and average number of people in the waiting queue per a particular demographic group, based on the demographic category determination from the facial images and analysis of the movement of the people from the person images.