Machine learning-based re-identification of shoppers in a cashier-less store for autonomous checkout
Systems and methods for tracking and re-identifying tracked subjects in an area of real space are disclosed. The method includes generating first and second reidentification feature vectors of a first subject identified from a first time interval from the respective first and second sequences of images as obtained from the first time interval. The method includes generating third and fourth reidentification feature vectors of a second subject identified from a second time interval from the respective first and second sequences of images. The method includes matching a second subject identified from a second time interval with the first subject identified from the first time interval when at least one of a first similarity score between the first and the third reidentification feature vectors and a second similarity score between the second and the fourth reidentification feature vectors is above a pre-defined threshold.
1. A method of re-identifying a previously identified subject in an area of real space, the method including:
receiving, from at least two cameras with overlapping fields of view, respective first and second sequences of images of corresponding fields of view in the area of real space;
generating first and second reidentification feature vectors of a first subject identified from a first time interval by performing operations including:
providing first and second images of the first subject from the respective first and second sequences of images and as obtained from the first time interval, to a trained machine learning model to produce respective first and second reidentification feature vectors; and
matching a second subject identified from a second time interval with the first subject identified from the first time interval by performing operations including:
providing third and fourth images of the second subject, from the respective first and second sequences of images and as obtained from the second time interval, to the trained machine learning model to produce respective third and fourth reidentification feature vectors;
calculating (i) a first similarity score between the first and the third reidentification feature vectors and (ii) a second similarity score between the second and the fourth reidentification feature vectors; and
re-identifying the second subject identified from the second time interval as the first subject identified from the first time interval when at least one of the first similarity score and the second similarity score is above a pre-defined threshold.
2. The method of claim 1 , wherein the re-identifying of the second subject identified from the second time interval, further includes:
calculating an average of the first similarity score and the second similarity score and re-identifying the second subject identified from the second time interval as the first subject identified from the first time interval when the average similarity score is above the pre-defined threshold.
3. The method of claim 1 , wherein the generating of the first, the second, the third, and the fourth reidentification feature vectors, further includes:
placing first, second, third, and fourth bounding boxes respectively around at least a portion of first, second, third, and fourth poses of the identified subjects, as identified from the first, the second, the third, and the fourth images of the respective sequences of images, to provide first, second, third, and fourth cropped out images as the first, the second, the third, and the fourth images.
4. The method of claim 3 , wherein the first pose, as identified from the first image, is identified from the first sequence of images from a first camera, the second pose, as identified from the second image, is identified from the second sequence of images from a second camera, the third pose, as identified from the third image, is identified from the first sequence of images from the first camera, and the fourth pose, as identified from the fourth image, is identified from the second sequence of images from the second camera.
5. The method of claim 3 , further including comparing a count of a number of identified subjects in the second time interval with a number of identified subjects in the first time interval and when the count of the number of identified subjects in the second time interval is less than the count of the number of identified subjects in the first time interval, performing a matching of the subjects identified in the second time interval with subjects identified in a time interval preceding the first time interval.
6. The method of claim 1 , wherein the first similarity score and the second similarity score are cosine similarity measures respectively between (i) the first and the third reidentification feature vectors and (ii) the second and the fourth reidentification feature vectors.
7. The method of claim 1 , further including identifying an error in tracking of the first subject identified from the first time interval, when the first similarity score and the second similarity score are below the pre-defined threshold.
8. The method of claim 7 , wherein the error is a single-swap error when the first subject from the first time interval is incorrectly matched to a third subject from the second time interval.
9. The method of claim 7 , wherein the error is a split error when the first subject from the first time interval is incorrectly matched to a fourth subject from the second time interval and wherein the fourth subject is identified in the second time interval.
10. The method of claim 7 , wherein the error is an enter-exit-swap error indicating that the second subject from the second time interval does not match the first subject from the first time interval and the second subject is a new subject who entered the area of real space in the second time interval and was not in the area of real space in the first time interval and the first subject is not present in the second time interval.
11. The method of claim 1 , wherein each of the respective first and second reidentification feature vectors represents learned visual features of the first subject and each of the respective third and fourth reidentification vectors represents learned visual features of the second subject.
12. The method of claim 1 , wherein the first, the second, the third and the fourth reidentification feature vectors further comprise first, second, third and fourth visual identifiers concatenated with respective first, second, third and fourth learned visual features, and wherein the visual identifiers represent at least a color of hair of the first subject and the second subject and a color of clothing of the first subject and the second subject captured from the first, the second, the third and the fourth images of the respective sequences of images.
13. The method of claim 1 , further including:
generating fifth and sixth reidentification feature vectors of a third subject identified from the first time interval by performing operations including:
providing fifth and sixth images of the third subject from the respective first and second sequences of images and as obtained from the first time interval, to the trained machine learning model to produce respective fifth and sixth reidentification feature vectors; and
matching the second subject identified from the second time interval with the first subject and the third subject identified from the first time interval by performing operations including:
calculating (i) a third similarity score between the between the fifth and the third reidentification feature vectors and (ii) a fourth similarity score between the sixth and the fourth reidentification feature vectors; and
re-identifying the second subject identified from the second time interval as the first subject identified from the first time interval when the third similarity score and the fourth similarity score are below the pre-defined threshold and when at least one of the first similarity score and the second similarity score is above the pre-defined threshold.
14. A system including one or more processors coupled to memory, the memory loaded with computer instructions to re-identify a previously identified subject in an area of real space, the instructions, when executed on the processors, implement actions comprising:
receiving, from at least two cameras with overlapping fields of view, respective first and second sequences of images of corresponding fields of view in the area of real space;
generating first and second reidentification feature vectors of a first subject identified from a first time interval by performing operations including:
providing first and second images of the first subject from the respective first and second sequences of images and as obtained from the first time interval, to a trained machine learning model to produce respective first and second reidentification feature vectors; and
matching a second subject identified from a second time interval with the first subject identified from the first time interval by performing operations including:
providing third and fourth images of the second subject, from the respective first and second sequences of images and as obtained from the second time interval, to the trained machine learning model to produce respective third and fourth reidentification feature vectors;
calculating (i) a first similarity score between the first and the third reidentification feature vectors and (ii) a second similarity score between the second and the fourth reidentification feature vectors; and
re-identifying the second subject identified from the second time interval as the first subject identified from the first time interval when at least one of the first similarity score and the second similarity score is above a pre-defined threshold.
15. The system of claim 14 , wherein the re-identifying of the second subject identified from the second time interval, further implementing actions comprising:
calculating an average of the first similarity score and the second similarity score and re-identifying the second subject identified from the second time interval as the first subject identified from the first time interval when the average similarity score is above the pre-defined threshold.
16. The system of claim 14 , wherein the generating of the first, the second, the third, and the fourth reidentification feature vectors, further implementing actions comprising:
placing first, second, third, and fourth bounding boxes respectively around at least a portion of first, second, third, and fourth poses of the identified subjects, as identified from the first, the second, the third, and the fourth images of the respective sequences of images, to provide first, second, third, and fourth cropped out images as the first, the second, the third, and the fourth images.
17. The system of claim 16 , wherein the first pose of the first subject from the first time interval is one of the at least a front pose, a side pose, and a back pose of the first subject from the first time interval.
18. A non-transitory computer readable storage medium impressed with computer program instructions to re-identify a previously identified subject in an area of real space, the instructions, when executed on a processor, implement a method comprising:
receiving, from at least two cameras with overlapping fields of view, respective first and second sequences of images of corresponding fields of view in the area of real space;
generating first and second reidentification feature vectors of a first subject identified from a first time interval by performing operations including:
providing first and second images of the first subject from the respective first and second sequences of images and as obtained from the first time interval, to a trained machine learning model to produce respective first and second reidentification feature vectors; and
matching a second subject identified from a second time interval with the first subject identified from the first time interval by performing operations including:
providing third and fourth images of the second subject, from the respective first and second sequences of images and as obtained from the second time interval, to the trained machine learning model to produce respective third and fourth reidentification feature vectors;
calculating (i) a first similarity score between the first and the third reidentification feature vectors and (ii) a second similarity score between the second and the fourth reidentification feature vectors; and
re-identifying the second subject identified from the second time interval as the first subject identified from the first time interval when at least one of the first similarity score and the second similarity score is above a pre-defined threshold.
19. The non-transitory computer readable storage medium of claim 18 , wherein the first similarity score and the second similarity score are cosine similarity measures respectively between (i) the first and the third reidentification feature vectors and (ii) the second and the fourth reidentification feature vectors.
20. The non-transitory computer readable storage medium of claim 18 , implementing the method further comprising:
generating fifth and sixth reidentification feature vectors of a third subject identified from the first time interval by performing operations including:
providing fifth and sixth images of the third subject from the respective first and second sequences of images and as obtained from the first time interval, to the trained machine learning model to produce respective fifth and sixth reidentification feature vectors; and
matching the second subject identified from the second time interval with the first subject and the third subject identified from the first time interval by performing operations including:
calculating (i) a third similarity score between the between the fifth and the third reidentification feature vectors and (ii) a fourth similarity score between the sixth and the fourth reidentification feature vectors; and
re-identifying the second subject identified from the second time interval as the first subject identified from the first time interval when the third similarity score and the fourth similarity score are below the pre-defined threshold and when at least one of the first similarity score and the second similarity score is below the pre-defined threshold.