Analyzing sensor data to identify events
This disclosure is directed to techniques in which a first user in an environment scans visual indicia associated with an item, such as a barcode, before handing the item to a second user. One or more computing devices may receive an indication of the scan, retrieve image data of the interaction from a camera within the environment, identify the user that received the item, and update a virtual cart associated with the second user to indicate addition of the item.
1. One or more computing devices comprising:
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:
receiving sensor data generated by a sensor in an environment, the sensor data identifying an item;
determining a portion of the environment associated with the sensor;
determining, from multiple cameras within the environment, a first camera having a field-of-view (FOV) that includes the portion of the environment associated with the sensor;
receiving image data generated by the first camera within the environment, the image data representing the portion of the environment associated with the sensor;
analyzing the image data;
identifying, based at least in part on the analyzing, a hand represented in the image data;
determining a user identifier associated with the hand; and
updating virtual-cart data associated with the user identifier to indicate an item identifier associated with the item.
2. The one or more computing devices as recited in claim 1 , wherein the receiving the sensor data generated by the sensor comprises receiving scan data generated by a scanning device that scanned visual indicia associated with the item.
3. The one or more computing devices as recited in claim 1 , wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising receiving data indicating a first time at which the sensor generated the sensor data, and wherein the analyzing comprises analyzing image data generated by the first camera after the first time and within a threshold amount of time of the first time.
4. The one or more computing devices as recited in claim 1 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor; and
the analyzing comprises analyzing at least a portion of the image data corresponding to the VOI.
5. The one or more computing devices as recited in claim 1 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises analyzing at least a portion of the image data corresponding to the VOI;
the identifying comprises identifying the hand after the hand enters the VOI.
6. The one or more computing devices as recited in claim 1 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises:
analyzing at least a portion of a first frame of the image data corresponding to the VOI; and
analyzing a least a portion of a second frame of the image data corresponding to the VOI;
the identifying comprises:
identifying, based at least in part on the analyzing of the at least the portion of the first frame, an empty hand within the VOI; and
identifying, based at least in part on the analyzing of the at least the portion of the second frame, a full hand within the VOI.
7. The one or more computing devices as recited in claim 1 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises:
analyzing at least a portion of a first frame of the image data corresponding to the VOI;
analyzing a least a portion of a second frame of the image data corresponding to the VOI;
analyzing a least a portion of a third frame of the image data corresponding to the VOI; and
analyzing a least a portion of a fourth frame of the image data corresponding to the VOI;
the identifying comprises:
identifying, based at least in part on the analyzing of the at least the portion of the first frame, an empty hand at a first location within the VOI; and
identifying, based at least in part on the analyzing of the at least the portion of the second frame, an empty hand at a second location within the VOI;
identifying, based at least in part on the analyzing of the at least the portion of the third frame, a full hand at a third location within the VOI; and
identifying, based at least in part on the analyzing of the at least the portion of the fourth frame, a full hand at a fourth location within the VOI;
the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising
determining a first direction vector based at least in part on the first location and the second location; and
determining a second direction vector based at least in part on the third location and the fourth location;
the updating comprises updating the virtual-cart data associated with the user identifier based at least in part on the first direction vector and the second direction vector.
8. The one or more computing devices as recited in claim 1 , wherein:
the analyzing comprises:
generating a segmentation map using a first frame of the image data, the segmentation map at least identifying a first set of pixels of the first frame corresponding to a hand of the user; and
inputting, into a trained classifier, first data indicating the first set of pixels of the first frame corresponding to the hand of the user;
the identifying comprises receiving, as output of the trained classifier, second data indicating that the hand received the item.
9. The one or more computing devices as recited in claim 1 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises analyzing at least a portion of the image data corresponding to the VOI;
the identifying comprises identifying the hand within the VOI; and
the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising identifying the item within the VOI.
10. A method comprising:
receiving sensor data generated by a sensor in an environment, the sensor data identifying an item;
determining a portion of the environment associated with the sensor;
determining, from multiple cameras within the environment, a first camera having a field-of-view (FOV) that includes the portion of the environment associated with the sensor;
receiving image data generated by the first camera within the environment, the image data representing the portion of the environment associated with the sensor;
analyzing the image data;
identifying, based at least in part on the analyzing, a hand represented in the image data;
determining a user identifier associated with the hand; and
updating virtual-cart data associated with the user identifier to indicate an item identifier associated with the item.
11. The method as recited in claim 10 , wherein the receiving the sensor data generated by the sensor comprises receiving scan data generated by a scanning device that scanned visual indicia associated with the item.
12. The method as recited in claim 10 , further comprising receiving data indicating a first time at which the sensor generated the sensor data, and wherein the analyzing comprises analyzing image data generated by the first camera after the first time and within a threshold amount of time of the first time.
13. The method as recited in claim 10 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor; and
the analyzing comprises analyzing at least a portion of the image data corresponding to the VOI.
14. The method as recited in claim 10 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises analyzing at least a portion of the image data corresponding to the VOI;
the identifying comprises identifying the hand after the hand enters the VOI.
15. The method as recited in claim 10 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises:
analyzing at least a portion of a first frame of the image data corresponding to the VOI; and
analyzing a least a portion of a second frame of the image data corresponding to the VOI;
the identifying comprises:
identifying, based at least in part on the analyzing of the at least the portion of the first frame, an empty hand within the VOI; and
identifying, based at least in part on the analyzing of the at least the portion of the second frame, a full hand within the VOI.
16. The method as recited in claim 10 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises:
analyzing at least a portion of a first frame of the image data corresponding to the VOI;
analyzing a least a portion of a second frame of the image data corresponding to the VOI;
analyzing a least a portion of a third frame of the image data corresponding to the VOI; and
analyzing a least a portion of a fourth frame of the image data corresponding to the VOI;
the identifying comprises:
identifying, based at least in part on the analyzing of the at least the portion of the first frame, an empty hand at a first location within the VOI; and
identifying, based at least in part on the analyzing of the at least the portion of the second frame, an empty hand at a second location within the VOI;
identifying, based at least in part on the analyzing of the at least the portion of the third frame, a full hand at a third location within the VOI; and
identifying, based at least in part on the analyzing of the at least the portion of the fourth frame, a full hand at a fourth location within the VOI;
the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising
determining a first direction vector based at least in part on the first location and the second location; and
determining a second direction vector based at least in part on the third location and the fourth location;
the updating comprises updating the virtual-cart data associated with the user identifier based at least in part on the first direction vector and the second direction vector.
17. The method as recited in claim 10 , wherein the analyzing comprises:
the analyzing comprises:
generating a segmentation map using a first frame of the image data, the segmentation map at least identifying a first set of pixels of the first frame corresponding to a hand of the user; and
inputting, into a trained classifier, first data indicating the first set of pixels of the first frame corresponding to the hand of the user;
the identifying comprises receiving, as output of the trained classifier, second data indicating that the hand received the item.
18. The method as recited in claim 10 , wherein:
the determining the portion comprises determining a volume of interest (VOI) within the environment relative to the sensor;
the analyzing comprises analyzing at least a portion of the image data corresponding to the VOI;
the identifying comprises identifying the hand within the VOI; and
the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising identifying the item within the VOI.
19. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:
receiving sensor data generated by a sensor in an environment, the sensor data identifying an item;
determining a portion of the environment associated with the sensor;
determining, from multiple cameras within the environment, a first camera having a field-of-view (FOV) that includes the portion of the environment associated with the sensor;
receiving image data generated by the first camera within the environment, the image data representing the portion of the environment associated with the sensor;
analyzing the image data;
identifying, based at least in part on the analyzing, a hand represented in the image data;
determining a user identifier associated with the hand; and
updating virtual-cart data associated with the user identifier to indicate an item identifier associated with the item.
20. The one or more non-transitory computer-readable media as recited in claim 19 , wherein the receiving the sensor data generated by the sensor comprises receiving scan data generated by a scanning device that scanned visual indicia associated with the item.