Systems and methods to check-in shoppers in a cashier-less store
Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.
1. A method for linking a subject in an area of real space with a user account, the method comprising:
performing a match by matching a mobile computing device of one or more mobile computing devices with a subject, the match being performed in dependence on a comparison of a determined velocity of the subject and determined velocities of the one or more mobile computing devices; and
linking, in dependence on the performed match, the subject with a user account corresponding to one mobile computing device of the mobile computing devices matched by the performed match.
2. The method of claim 1 , further comprising determining the velocities of the one or more mobile computing devices by implementing operations including:
receiving first accelerometer data from a first mobile computing device of the one or more mobile computing devices, the first accelerometer data indicative of acceleration of the first mobile computing device along one or more axes;
using the first accelerometer data from the first mobile computing device over a plurality of time intervals to determine a corresponding plurality of velocities of the first mobile computing device over the plurality of time intervals; and
determining an average velocity of the first mobile computing device over the plurality of time intervals, based on the plurality of velocities.
3. The method of claim 1 , further comprising determining the velocity of the subject by implementing operations including:
determining a change in a position of a joint of the subject with respect to time in a sequence of images of the area of real space, wherein the subject is identified within the sequence of images;
determining a location of a center of mass of the subject, based on the joint of the subject; and
determining a velocity of the center of mass of the subject, based on the change in the position of the joint of the subject with respect to time.
4. The method of claim 1 , further comprising determining the velocity of the subject by implementing operations including:
determining a velocity of a hand of the subject holding a mobile computing device.
5. The method of claim 1 , wherein the performing of the match includes:
determining Euclidean distances between velocities of the one or more mobile computing devices and the subject; and
in response to a Euclidean distance between a velocity of a particular mobile computing device and the subject being less than a threshold, matching the particular mobile computing device with the subject.
6. The method of claim 1 , wherein the performing of the match includes:
determining Euclidean distances between velocities of various pairs of mobile computing devices and the subject, such that a plurality of Euclidean distances are determined corresponding to a plurality of pairs of mobile computing devices and the subject, each pair including (i) a corresponding mobile computing device of the one or more mobile computing devices that have not yet been linked to the subject and (ii) the subject that has not yet been linked to a corresponding one or more mobile computing device;
for each pair of the plurality of pairs, incrementing a score counter in response to a corresponding Euclidean distance associated with the pair being below a first threshold, such that a plurality of score counters corresponding to the plurality of pairs is maintained and selectively incremented;
comparing individual score counters with a second threshold, and selecting (i) a highest-score counter with a highest score above the second threshold and (ii) a second-highest score counter with a second highest score, wherein the highest-score counter is for a first pair that includes a first mobile computing device and the subject; and
in response to a difference between the highest-score counter for the first pair and the second- highest score counter being higher than a third threshold, matching the first mobile computing device with the subject.
7. The method of claim 6 , wherein the performing of the match further comprises incrementing a score counter for a pair of the mobile computing device and the subject when the distance between the velocity of the mobile computing device and the velocity of the subject is below a first threshold.
8. The method of claim 7 , further including:
comparing score counters for multiple pairs of mobile computing devices the subject with a second threshold and selecting a score counter with a highest score above the second threshold;
comparing the score of the selected score counter with a score of a score counter with a second highest score; and
matching a pair of a mobile computing device and the subject with the selected score counter when a difference between the scores of the selected score counter and the score of the score counter with the second highest score is above a third threshold.
9. The method of claim 8 , further including using a trained neural network to predict the first threshold, the second threshold and the third threshold.
10. The method of claim 6 , further including calculating the velocity of the subject from a change in a center of mass of the subject over time, wherein the center of mass of the subject is determined from a location of a joint of the subjects.
11. The method of claim 6 , further including processing a plurality of sequences of images to determine a hand joint of the subject holding the mobile computing device and calculating a velocity of the hand joint over time.
12. A non-transitory computer readable storage medium impressed with computer program instructions to link a subject in an area of real space with a user account, the instructions, when executed on a processor, implement a method comprising:
performing a match by matching a mobile computing device of one or more mobile computing devices with a subject, the match being performed in dependence on a comparison of a determined velocity of the subject and determined velocities of the one or more mobile computing devices; and
linking, in dependence on the performed match, the subject with a user account corresponding to one mobile computing device of the mobile computing devices matched by the performed match.
13. The non-transitory computer readable storage medium of claim 12 , wherein the method further comprises:
receiving sensor data from the area of real space, wherein the sensor data comprises one or more sequences of image data respectively generated by one or more cameras;
determining velocities of one or more subjects, based on the sensor data;
determining velocities of the one or more mobile computing devices in the area of real space, based on data transmitted by the one or more mobile computing devices;
determining a norm between (i) a velocity of a first mobile computing device of the one or more mobile computing devices and (ii) a velocity of a first subject of the one or more subjects; and
in response to the norm being less than a threshold, linking the first subject with a first user account linked to a first client application being executed on the first mobile device.
14. The non-transitory computer readable storage medium of claim 12 , wherein the method further comprises determining the velocities of the one or more mobile computing devices by implementing operations including:
receiving first accelerometer data from a first mobile computing device of the one or more mobile computing devices, the first accelerometer data indicative of acceleration of the first mobile computing device along one or more axes;
using the first accelerometer data from the first mobile computing device over a plurality of time intervals to determine a corresponding plurality of velocities of the first mobile computing device over the plurality of time intervals; and
determining an average velocity of the first mobile computing device over the plurality of time intervals, based on the plurality of velocities.
15. The non-transitory computer readable storage medium of claim 12 , wherein the method further comprises determining the velocity of the subject by implementing operations including:
determining a change in a position of a joint of the subject with respect to time in a sequence of images of the area of real space, wherein the subject is identified within the sequence of images;
determining a location of a center of mass of the subject, based on the joint of the subject; and
determining a velocity of the center of mass of the subject, based on the change in the position of the joint of the subject with respect to time.
16. A system including one or more processors coupled to memory, the memory storing computer instructions for linking a subject in an area of real space with a user account, the computer instructions, when executed on the one or more processors, causing the one or more processors to implement actions comprising:
performing a match by matching a mobile computing device of one or more mobile computing devices with the subject, the match being performed in dependence on a comparison of a determined velocity of a subject and determined velocities of the one or more mobile computing devices; and
linking, in dependence on the performed match, the subject with a user account corresponding to one mobile computing device of the mobile computing devices matched by the performed match.
17. The system of claim 16 , wherein the actions further comprise determining the velocities of the one or more mobile computing devices by implementing operations including:
receiving first accelerometer data from a first mobile computing device of the one or more mobile computing devices, the first accelerometer data indicative of acceleration of the first mobile computing device along one or more axes;
using the first accelerometer data from the first mobile computing device over a plurality of time intervals to determine a corresponding plurality of velocities of the first mobile computing device over the plurality of time intervals; and
determining an average velocity of the first mobile computing device over the plurality of time intervals, based on the plurality of velocities.
18. The system of claim 16 , wherein the actions further comprise determining the velocity of the subject by implementing operations including:
determining a change in a position of a joint of the subject with respect to time in a sequence of images of the area of real space, wherein the subject is identified within the sequence of images;
determining a location of a center of mass of the subject, based on the joint of the subject; and
determining a velocity of the center of mass of the subject, based on the change in the position of the joint of the subject with respect to time.
19. The system of claim 16 , wherein the actions further comprise determining the velocity of the subject by implementing operations including:
determining a velocity of a hand of the subject holding a mobile computing device.