Method and system for detecting shopping groups based on trajectory dynamics
View Patent ↗The present invention is a method and system for detecting shopping groups based on the dynamic relation between shoppers' trajectories. First, shopper trajectories are generated using video images or positioning devices. Then, group behavior features are extracted from a candidate shopping group trajectory pair. The group behavior features of a given pair of trajectories are typically the changes in positional differences and the average speed of these trajectories. From a model of shopping group behavior, a given pair of candidate shopping group trajectories is analyzed to determine the group score—the likelihood of the pair of shoppers indeed belonging to the same group. Lastly, the system utilizes graph segmentation framework to find clusters where trajectories belonging to each cluster have tight group scores with each other.
1. A method for detecting shopping groups based on dynamic relation between shoppers' trajectories, comprising:
a) building a system of video and wireless signal sensors to collect images and locations of shoppers across multiple fields of view;
b) setting up a shopper group behavior model, wherein the shopper group behavior model consists of behavior primitive states of related shoppers, and wherein the shopper group behavior model represents all possible progressions of states when two related shoppers are in proximity;
c) generating shoppers' trajectories from images captured by at least one video sensor or at least one wireless signal sensor, the trajectories generated using a shopper trajectory estimation module that performs the following steps:
i) segmenting a motion foreground from a camera view,
ii) detecting a shopper from the camera view,
iii) tracking the shopper across multiple frames, and
iv) estimating the trajectory based on the tracking of the shopper;
d) extracting group behavior features from pairs of the trajectories, the group behavior features including changes in distances between the pairs of the trajectories and average speeds of the pairs of the trajectories,
wherein the changes in distances between the pairs of the trajectories are determined by computing a difference trajectory between each pair of trajectories in the pairs of the trajectories,
wherein the difference trajectory is computed for a time period during which two trajectories overlap, and
wherein the difference trajectory consists of points that are vector differences between points from a first trajectory and points from a second trajectory at each matching time instance;
e) analyzing the group behavior features based on the shopper group behavior model to determine group scores, wherein each of the group scores quantizes the likelihood of the corresponding pair of shoppers belonging to a shopping group;
f) segmenting the trajectories into shopping groups based on the group scores;
g) merging multiple shopping groups into a single shopping group using a multi-trajectory shopper model to determine interactions among the trajectories of multiple shoppers;
h) utilizing demographic information of the shoppers to adjust the group scores, wherein demographic categories, including gender, age, or ethnicity, of the shoppers constrain relations among the shoppers; and
i) generating output data for further analysis, wherein the output data comprises group composition, group dynamics and group performance,
wherein group composition comprises the number, gender, and age compositions of people in a group, and
wherein the group dynamics comprise identification of a group leader by their trajectory identification and a percentage of time spent together.
2. The method according to claim 1 , wherein the behavior primitive states of the shopper group behavior model comprise staying together, walking together, merging, and splitting, and
wherein state progression of a pair of the trajectories is determined using transitions among states based on the modeled relationship between the states and the group behavior features.
3. The method according to claim 1 , wherein the method further comprises a step of utilizing position based prior to adjust the group scores, wherein the position based prior discounts a high group score for a pair of the trajectories and reduces the admissible range for a correct grouping when the location of the pair of the trajectories is busy or when the trajectories are close to each other due to physical constraints at the location.
4. The method according to claim 1 , wherein the multi-trajectory shopper model involves three or more shoppers and examines whether any two shopping groups, including a shopping group that has only one shopper, belong to a single shopping group.
5. The method according to claim 1 , wherein the group performance comprises shopping duration of a detected shopping group and total money spent by the detected shopping group.
6. A method for detecting shopping groups based on dynamic relation between shoppers' trajectories, comprising:
a) building a system of video and wireless signal sensors to collect images and locations of shoppers across multiple fields of view;
b) setting up a non-group behavior model, wherein the non-group behavior model consists of behavior primitive states of unrelated shoppers, and wherein the non-group behavior model represents all possible progressions of states when two unrelated shoppers are in proximity;
c) generating shoppers' trajectories from images captured by at least one video sensor or at least one wireless signal sensor, the trajectories generated using a shopper trajectory estimation module that performs the following steps:
i) segmenting a motion foreground from a camera view,
ii) detecting a shopper from the camera view,
iii) tracking the shopper across multiple frames, and
iv) estimating the trajectory based on the tracking of the shopper;
d) extracting group behavior features from pairs of the trajectories, the group behavior features including changes in distances between the pairs of the trajectories and average speeds of the pairs of the trajectories,
wherein the changes in distances between the pairs of the trajectories are determined by computing a difference trajectory between each pair of trajectories in the pairs of the trajectories,
wherein the difference trajectory is computed for a time period during which two trajectories overlap, and
wherein the difference trajectory consists of points that are vector differences between points from a first trajectory and points from a second trajectory at each matching time instance;
e) analyzing the group behavior features based on the non-group behavior model to determine group scores, wherein each of the group scores quantizes the likelihood of the corresponding pair of shoppers belonging to a shopping group;
f) segmenting the trajectories into shopping groups based on the group scores;
g) merging multiple shopping groups into a single shopping group using a multi-trajectory shopper model to determine interactions among the trajectories of multiple shoppers;
h) utilizing demographic information of the shoppers to adjust the group scores, wherein demographic categories, including gender, age, or ethnicity, of the shoppers constrain relations among the shoppers; and
i) generating output data for further analysis, wherein the output data comprises group composition, group dynamics, and group performance,
wherein group composition comprises the number, gender, and age compositions of people in a detected shopping group, and
wherein the group dynamics comprise identification of a group leader by their trajectory identification and a percentage of time spent together.
7. The method according to claim 6 , wherein the behavior primitive states of the non-group behavior model comprise staying together, crossing, merging, and splitting, and
wherein state progression of a pair of the trajectories is determined using transitions among states based on the modeled relationship between the states and the group behavior features.
8. The method according to claim 6 , wherein the method further comprises a step of utilizing position based prior to adjust the group scores, wherein the position based prior discounts a high group score for a pair of the trajectories and reduces the admissible range for a correct grouping when the location of the pair of the trajectories is busy or when the trajectories are close to each other due to physical constraints at the location.
9. The method according to claim 6 , wherein the multi-trajectory shopper model involves three or more shoppers and examines whether any two shopping groups, including a shopping group that has only one shopper, belong to a single shopping group.
10. The method according to claim 6 , wherein the group performance comprises shopping duration of a detected shopping group and total money spent by the detected shopping group.