Using sensor data to determine activity
This disclosure is directed to, in part, a processing pipeline for detecting predefined activity using image data, identifying an item-of-interest and a location of the item-of-interest across frames of the image data, determining a trajectory of the item-of-interest, and determining an identifier of the item-of-interest and an action taken with respect to the item-of-interest. The processing pipeline may utilize one or more trained classifiers and, in some instances, additional data to identify items that are placed into or removed from a tote (e.g., basket, cart, or other receptacle) by users in material handling facilities as the users move around the material handling facilities.
1. A method comprising:
analyzing a first set of multiple frames of image data to determine a first trajectory of a first item across the first set of multiple frames;
determining a first item identifier associated with the first item;
adding the first item identifier to a virtual cart based at least in part on the first trajectory;
analyzing a second set of multiple frames of image data to determine a second trajectory of a second item across the second set of multiple frames;
determining a second item identifier associated with the second item; and
storing an indication that an action associated with the second item identifier is unknown based at least in part on the second trajectory.
2. The method as recited in claim 1 , wherein the first trajectory of the first item comprises a trajectory towards a receptable of a mobile cart.
3. The method as recited in claim 1 , wherein the determining the first item identifier comprises determining the first item identifier based at least in part on the analyzing of the first set of multiple frames of image data.
4. The method as recited in claim 1 , wherein the determining the first item identifier comprises receiving an indication of the first item identifier from a sensor coupled to a mobile cart.
5. The method as recited in claim 1 , further comprising:
analyzing a third set of multiple frames of image data to determine a third trajectory of the first item across the third set of multiple frames;
determining the first item identifier associated with the first item; and
removing the first item identifier from the virtual cart based at least in part on the third trajectory.
6. The method as recited in claim 5 , wherein the first trajectory of the first item comprises a trajectory away from a receptacle of a mobile cart.
7. The method as recited in claim 1 , wherein the analyzing the first set of multiple frames of image data comprises analyzing the first set of multiple frames of image data generated by one or more cameras coupled to a mobile cart.
8. The method as recited in claim 1 , wherein the analyzing the first set of multiple frames of image data comprises analyzing the first set of multiple frames of image data generated by one or more overhead cameras in a facility.
9. The method as recited in claim 1 , further comprising:
inputting trajectory data associated with the first trajectory into a classifier; and
receiving, as output of the classifier, data indicating the first item identifier and an action taken with respect to the first item, the action indicating that the first item was placed in a receptacle of a mobile cart;
and wherein the adding comprises adding the first item identifier to the virtual cart based at least in part on the action indicating that the first item was placed in the receptacle of the mobile cart.
10. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
analyzing a first set of multiple frames of image data to determine a first trajectory of a first item across the first set of multiple frames;
determining a first item identifier associated with the first item;
adding the first item identifier to a virtual cart based at least in part on the first trajectory;
analyzing a second set of multiple frames of image data to determine a second trajectory of a second item across the second set of multiple frames;
determining a second item identifier associated with the second item; and
storing an indication that an action associated with the second item identifier is unknown based at least in part on the second trajectory.
11. The system as recited in claim 10 , wherein the first trajectory of the first item comprises a trajectory towards a receptable of a mobile cart.
12. The system as recited in claim 10 , wherein the determining the first item identifier comprises determining the first item identifier based at least in part on the analyzing of the first set of multiple frames of image data.
13. The system as recited in claim 10 , wherein the determining the first item identifier comprises receiving an indication of the first_item identifier from a sensor coupled to a mobile cart.
14. The system as recited in claim 10 , the operations further comprising:
analyzing a third set of multiple frames of image data to determine a second third trajectory of the first item across the third set of multiple frames;
determining the first item identifier associated with the first item; and
removing the first item identifier from the virtual cart based at least in part on the third trajectory.
15. The system as recited in claim 14 , wherein the first trajectory of the first item comprises a trajectory away from a receptacle of a mobile cart.
16. The system as recited in claim 10 , wherein the analyzing the first set of multiple frames of image data comprises analyzing the first set multiple frames of image data generated by one or more cameras coupled to a mobile cart.
17. The system as recited in claim 10 , wherein the analyzing the first set of multiple frames of image data comprises analyzing the first set of multiple frames of image data generated by one or more overhead cameras in a facility.
18. The system as recited in claim 10 , operations further comprising:
inputting trajectory data associated with the first trajectory into a classifier; and
receiving, as output of the classifier, data indicating the first item identifier and an action taken with respect to the first item, the action indicating that the first item was placed in a receptacle of a mobile cart;
and wherein the adding comprises adding the first item identifier to the virtual cart based at least in part on the action indicating that the first item was placed in the receptacle of the mobile cart.
19. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform operations comprising:
analyzing a first set of multiple frames of image data to determine a first trajectory of a first item across the first set of multiple frames;
determining a first item identifier associated with the first item;
adding the first item identifier to a virtual cart based at least in part on the first trajectory;
analyzing a second set of multiple frames of image data to determine a second trajectory of a second item across the second set of multiple frames;
determining a second item identifier associated with the second item; and
storing an indication that an action associated with the second item identifier is unknown based at least in part on the second trajectory.
20. The one or more non-transitory computer-readable media as recited in claim 19 , wherein the first trajectory of the first item comprises a trajectory towards a receptable of a mobile cart.