Automatic detection and aggregation of demographics and behavior of people
View Patent ↗The present invention is a system and framework for automatically measuring and correlating visual characteristics of people and accumulating the data for the purpose of demographic and behavior analysis. The demographic and behavior characteristics of people are extracted from a sequence of images using techniques from computer vision. The demographic and behavior characteristics are combined with a timestamp and a location marker to provide a feature vector of a person at a particular time at a particular location. These feature vectors are then accumulated and aggregated automatically in order to generate a data set that can be statistically analyzed, data mined and/or queried.
1. A method for detecting and aggregating demographics and behavior of people, comprising the following steps of:
a) detecting the people from a plurality of input images in the vicinity of a fixture with at least a visual sensing device using a control and processing system,
wherein the visual sensing device is connected to a means for video interface,
b) detecting faces in the plurality of input images,
c) detecting locations of facial features on the detected faces and estimating facial geometry including position, size, and orientation of the detected faces,
d) aligning the detected faces using the facial geometry estimation,
e) building appearance models of the people across image frames containing the detected faces, by averaging pixel values of the detected faces, aligned using estimates for the position, size, and orientation of the detected faces,
wherein an estimation of the position, size, and orientation is carried out by employing a parallel array of multiple learning machine regressors,
f) tracking the detected faces by keeping identities assigned to the detected faces,
g) obtaining demographics information and behavior features of the people automatically,
h) computing timestamps and location markers for the obtained demographics information and behavior features, and
i) combining the obtained demographics information and behavior features with the timestamps and the location markers into a feature vector,
wherein the behavior is detected based on an analysis of measurements for the behavior features,
wherein the behavior features comprise timestamps, start time and end time of tracks for appearance and disappearance of the detected faces, duration of watching fixtures, counts of the detected faces, counts of attentive faces, viewership, location markers, and facial pose estimates, and
wherein the demographics information comprises estimates of gender, age, and ethnicity of the people.
2. The method according to claim 1 , wherein the method further comprises a step of estimating poses of the people by using a plurality of learning machines, each of which is specialized to a given pose range.
3. The method according to claim 1 , wherein the method further comprises a step of managing face tracks to find a correct match between a history of face tracks and a new input face, using a geometric match score and an appearance match score,
wherein the geometric match score measures a difference in position, size, and time between a corrected face and a last face in the face tracking, and
wherein the appearance match score measures a difference between the corrected face and an average face appearance stored in the face tracking.
4. The method according to claim 1 , wherein the method further comprises a step of automatically classifying demographics of the people based on demographics scores of the aligned faces,
wherein two-dimensional poses of the detected faces are corrected so that the facial features across the detected faces are aligned.
5. An apparatus for detecting and aggregating demographics and behavior of people, comprising:
a) at least a control and processing system programmed to perform the following steps of:
detecting the people from a plurality of input images in the vicinity of a fixture with at least a visual sensing device,
wherein the visual sensing device is connected to a means for video interface,
detecting faces in the plurality of input images,
detecting locations of facial features on the detected faces and estimating facial geometry including position, size, and orientation of the detected faces,
aligning the detected faces using the facial geometry estimation,
building appearance models of the people across image frames containing the detected faces, by averaging pixel values of the detected faces, aligned using estimates for the position, size, and orientation of the detected faces,
wherein an estimation of the position, size, and orientation is carried out by employing a parallel array of multiple learning machine regressors,
tracking the detected faces by keeping identities assigned to the detected faces,
obtaining demographics information and behavior features of the people automatically,
computing timestamps and location markers for the obtained demographics information and behavior features, and
combining the obtained demographics information and behavior features with the timestamps and the location markers into a feature vector, and
b) at least a means for storing data,
wherein the behavior is detected based on an analysis of measurements for the behavior features,
wherein the behavior features comprise timestamps, start time and end time of tracks for appearance and disappearance of the detected faces, duration of watching fixtures, counts of the detected faces, counts of attentive faces, viewership, location markers, and facial pose estimates, and
wherein the demographics information comprises estimates of gender, age, and ethnicity of the people.
6. The apparatus according to claim 5 , wherein the apparatus further comprises a control and processing system for estimating poses of the people by using a plurality of learning machines, each of which is specialized to a given pose range.
7. The apparatus according to claim 5 , wherein the apparatus further comprises a control and processing system for managing face tracks to find a correct match between a history of face tracks and a new input face, using a geometric match score and an appearance match score,
wherein the geometric match score measures a difference in position, size, and time between a corrected face and a last face in the face tracking, and
wherein the appearance match score measures a difference between the corrected face and an average face appearance stored in the face tracking.
8. The apparatus according to claim 5 , wherein the apparatus further comprises a control and processing system for automatically classifying demographics of the people based on demographics scores of the aligned faces,
wherein two-dimensional poses of the detected faces are corrected so that the facial features across the detected faces are aligned.