Method and system for finding correspondence between point-of-sale data and customer behavior data
The present invention is a method and system to provide correspondences between point-of-sale data registered at the store checkout and shopper behavior data observed at point-of-purchase through video analysis. The point-of-sale data include the list of shoppers and purchase items, and the shopper behavior data include the purchase events along with observed purchase items. The correspondence in the form of checkout shopper IDs matched to purchase event IDs is derived based on the algebraic constraint among the point-of-sale data and the purchase event data. Additional constraint based on shopper tracks and checkout/event times can also be incorporated to the correspondence problem. Uncertainties due to the video measurement can be systematically handled utilizing a Bayesian model.
1. A method for finding correspondence between retail point-of-sale data and shopper behavior data along with shopper track data from video observation, comprising the following steps of:
a) constructing a relational database of purchase items versus shoppers from the point-of-sale data,
b) collecting images of shoppers using cameras with fields of view at point-of-purchase,
c) triggering proximity and foreground object analysis of the images based on shopper orientation and time-based behavior detection,
d) running video analysis on the images to track the body and hands of shoppers using a computer,
e) calculating the shopper behavior data based on information from tracking the shoppers, processed by the shopper behavior analysis module,
f) constructing a relational database of purchase items versus purchase events from the shopper behavior data,
g) incorporating event association constraints,
h) incorporating event time constraints using a linear equation, wherein the linear equation comprises a matrix that encodes relations between the purchase events and shoppers and a matrix that encodes time relations between the purchase events and checkout events, wherein the linear equation is
Min
ep
[
∑
ij
{
ip
ij
-
∑
k
ie
ik
ep
kj
}
2
+
α
∑
j
DistFit
j
+
β
∑
j
LastStopFit
j
+
λ
∑
j
VisualFit
j
wherein
Min
ep
[
∑
ij
{
ip
ij
-
∑
k
ie
ik
ep
kj
}
2
represents an optimization problem for determining a likely event-person matrix,
α
∑
j
DistFit
j
represents a constraint of distance between items,
β
∑
j
LastStopFit
j
represents a constraint of distance between a closest item in a basket to a checkout register, and
λ
∑
j
VisualFit
j
represents a constraint of the appearance of a shopper,
i) setting up a relational database of purchase events versus shoppers as unknown variables representing a correspondence,
j) building a correspondence equation based on the relational database of purchase items versus shoppers, the relational database of purchase items versus purchase events, and the relational database of purchase events versus shoppers, and the shopper track data,
k) solving the correspondence equation to find correspondence data, and
l) displaying correspondence data on a display screen to analyze the relationship the retail point-of-sale data and the shopper behavior data for the purpose of a media element rating system.
2. The method according to claim 1 , wherein the method further comprises a step of constructing the relational database of purchase items versus shoppers using a matrix,
wherein each row of the matrix represents a shopper, each column of the matrix represents a purchase item, and a non-zero value in an item and shopper entry of the matrix represents the purchase of the item by the shopper.
3. The method according to claim 1 , wherein the method further comprises a step of constructing the relational database of purchase items versus purchase events using a matrix,
wherein each row of the matrix represents a purchase item, each column of the matrix represents a purchase event, and a non-zero value in an item and event entry of the matrix represents the purchase of the item at the purchase event.
4. The method according to claim 1 , wherein the method further comprises a step of setting up data from the relational database of purchase events versus shoppers as unknown variables,
wherein the variables comprise a matrix where each row of the matrix represents a purchase event, each column of the matrix represents a shopper, and a non-zero value in an event and shopper entry of the matrix represents the occurrence of the purchase event caused by the shopper.
5. The method according to claim 1 , wherein the method further comprises a step of constructing a matrix that constrains event time, wherein each row of the matrix represents a shopper and zero elements in the row represent start time and end time of the purchase events.
6. The method according to claim 1 , wherein the method further comprises a step of constructing a matrix that constrains event time, wherein a permissible time range depends on the number of checkout items, wherein the number of checkout items is algebraically represented as
NumItem
j
=
∑
i
ip
ij
,
wherein NumItem j is the number of checkout items and
∑
i
ip
ij
represents the sum of items associated with a single shopper.
7. The method according to claim 1 , wherein the method further comprises a step of incorporating event association constraints using a matrix, wherein each entry of the matrix represents a relation between corresponding purchase events based on the shopper track data.
8. The method according to claim 1 , wherein the method further comprises a step of incorporating event association constraints using a quadratic form, wherein the quadratic form comprises a matrix that encodes relations between the purchase events and shoppers and a matrix that encodes associations among purchase events by shopper tracks.
9. The method according to claim 1 , wherein the method further comprises a step of utilizing probability distributions to represent data in the relational database and uncertainties in the data, utilizing a Bayesian formula to compute a posterior probability density of shoppers given observed purchase items.
10. The method according to claim 9 , wherein the method further comprises a step of utilizing probabilistic likelihood of purchase items purchased by the shoppers to represent the relational database of purchase items versus shoppers, wherein the probabilistic likelihood is represented as P(S t |I t ), wherein S t represents a shopper and I t represents an item.
11. The method according to claim 9 , wherein the method further comprises a step of utilizing probabilistic likelihood of true purchase items given observed purchase items at the purchase events to represent the relational database of purchase items versus purchase events wherein the probabilistic likelihood is represented as P(OH f |OI t , OI u ) wherein OI f , OI t , OI u are observed purchase items.
12. The method according to claim 9 , wherein the method further comprises a step of utilizing probabilistic likelihood of true purchase items given observed purchase items that is computed from a historical data of true purchase items and observed purchase items, wherein the probabilistic likelihood is represented as P(I t |OI t ) wherein I t is a true purchase item and OI t is an observed true purchase item.
13. The method according to claim 9 , wherein the method further comprises a step of utilizing probabilistic likelihood of checkout events and purchase events belonging to a same shopper as an event time constraint.
14. The method according to claim 9 , wherein the method further comprises a step of utilizing probability that any pair of purchase events belong to a same shopper as an event association constraint, wherein the probabilistic likelihood is represented as P(S t =S u ), wherein S t is a shopper of a first event and S u is a shopper of a second event.
15. An apparatus for finding correspondence between retail point-of-sale data and shopper behavior data along with shopper track data from video observation, comprising:
a) video cameras at point-of-purchase for collecting images of shoppers in each field of view,
b) a computer and a shopper behavior analysis module that perform the following steps of:
i. constructing a relational database of purchase items versus shoppers from the point-of-sale data,
ii. collecting images of shoppers using cameras with fields of view at point-of-purchase,
iii. triggering proximity and foreground object analysis of the images based on shopper orientation and time-based behavior detection,
iv. running video analysis on the images to track the body and hands of shoppers,
v. calculating the shopper behavior data based on information from tracking the shoppers, processed by the shopper behavior analysis module,
vi. constructing a relational database of purchase items versus purchase events from the shopper behavior data,
vii. incorporating event association constraints,
viii. incorporating event time constraints using a linear equation, wherein the linear equation comprises a matrix that encodes relations between the purchase events and shoppers and a matrix that encodes time relations between the purchase events and checkout events, wherein the linear equation is
Min
ep
[
∑
ij
{
ip
ij
-
∑
k
ie
ik
ep
kj
}
2
+
α
∑
j
DistFit
j
+
β
∑
j
LastStopFit
j
+
λ
∑
j
VisualFit
j
wherein
Min
ep
[
∑
ij
{
ip
ij
-
∑
k
ie
ik
ep
kj
}
2
represents an optimization problem for determining a likely event-person matrix,
α
∑
j
DistFit
j
represents a constraint of distance between items,
β
∑
j
LastStopFit
j
represents a constraint of distance between a closest item in a basket to a checkout register, and
λ
∑
j
VisualFit
j
represents a constraint of the appearance of a shopper,
ix. setting up a relational database of purchase events versus shoppers as unknown variables representing a correspondence,
x. building a correspondence equation based on the relational database of purchase items versus shoppers, the relational database of purchase items versus purchase events, the relational database of purchase events versus shoppers, and the shopper track data,
xi. solving the correspondence equation to find the correspondence, and
xii. displaying correspondence data on a display screen to analyze the relationship the retail point-of-sale data and the shopper behavior data for the purpose of a media element rating system.
16. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of constructing the relational database of purchase items versus shoppers using a matrix,
wherein each row of the matrix represents a shopper, each column of the matrix represents a purchase item, and a non-zero value in an item and shopper entry of the matrix represents the purchase of the item by the shopper.
17. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of constructing the relational database of purchase items versus purchase events using a matrix,
wherein each row of the matrix represents a purchase item, each column of the matrix represents a purchase event, and a non-zero value in an item and event entry of the matrix represents the purchase of the item at the purchase event.
18. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of setting up data from the relational database of purchase events versus shoppers as unknown variables,
wherein the variables comprise a matrix where each row of the matrix represents a purchase event, each column of the matrix represents a shopper, and a non-zero value in an event and shopper entry of the matrix represents the occurrence of the purchase event caused by the shopper.
19. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of constructing a matrix that constrains event time, wherein each row of the matrix represents a shopper and zero elements in the row represent start time and end time of the purchase events.
20. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of constructing a matrix that constrains event time, wherein a permissible time range depends on the number of checkout items, wherein the number of checkout items is algebraically represented as
NumItem
j
=
∑
i
ip
ij
,
wherein NumItem j is the number of checkout items and
∑
i
ip
ij
represents the sum of items associated with a single shopper.
21. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of incorporating event association constraints using a matrix, wherein each entry of the matrix represents a relation between corresponding purchase events based on the shopper track data.
22. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of incorporating event association constraints using a quadratic form, wherein the quadratic form comprises a matrix that encodes relations between the purchase events and shoppers and a matrix that encodes associations among purchase events by shopper tracks.
23. The apparatus according to claim 15 , wherein the apparatus further comprises a computer that performs a step of utilizing probability distributions to represent data in the relational database and uncertainties in the data, utilizing Bayesian formula to compute the posterior probability density of shoppers given observed purchase items.
24. The apparatus according to claim 23 , wherein the apparatus further comprises a computer that performs a step of utilizing probabilistic likelihood of purchase items purchased by the shoppers to represent the relational database of purchase items versus shoppers, wherein the probabilistic likelihood is represented as P(S t |I t ), wherein S t represents a shopper and I t represents an item.
25. The apparatus according to claim 23 , wherein the apparatus further comprises a computer that performs a step of utilizing probabilistic likelihood of true purchase items given observed purchase items at the purchase events to represent the relational database of purchase items versus purchase events.
26. The apparatus according to claim 23 , wherein the apparatus further comprises a computer that performs a step of utilizing probabilistic likelihood of true purchase items given observed purchase items that is computed from a historical data of true purchase items and observed purchase items, wherein the probabilistic likelihood is represented as P(I t |OI t ) wherein I T is a true purchase item and OI t is an observed true purchase item.
27. The apparatus according to claim 23 , wherein the apparatus further comprises a computer that performs a step of utilizing probabilistic likelihood of checkout events and purchase events belonging to a same shopper as an event time constraint.
28. The apparatus according to claim 23 , wherein the apparatus further comprises a computer that performs a step of utilizing probability that any pair of purchase events belong to a same shopper as an event association constraint, wherein the probabilistic likelihood is represented as P(S t =S u ), wherein S t is a shopper of a first event and S u is a shopper of a second event.