Inventory tracking system and method that identifies gestures of subjects holding inventory items
A method for identifying gestures in an area of real space is provided. The method including using a plurality of sensors to produce respective sequences of frames of corresponding fields of view in the area of real space, detecting subjects in the area of real space, using a sequence of frames produced by a corresponding sensor in the plurality of sensors in a first inference engine to identify inventory items carried by the detected subjects in the sequence of frames, using outputs of the first inference engine over a period of time in a second inference engine to identify gestures of the detected subjects, and storing the identified gestures in a database.
1. A method for identifying gestures in an area of real space, the method including:
using a plurality of sensors to produce respective sequences of frames of corresponding fields of view in the area of real space;
detecting subjects in the area of real space;
during a first production phase that implements a first inference engine that is pre-trained to operate in a first production mode, switching the first inference engine from a first training mode to the first production mode to identify inventory items carried by the detected subjects in the sequence of frames, wherein the first production phase includes using a sequence of frames produced by a corresponding sensor in the plurality of sensors to identify the inventory items carried by the detected subjects;
during a second production phase that implements a second inference engine that is pre-trained to operate in a second production mode, switching the second inference engine from a second training mode to the second production mode to identify gestures of the detected subjects carrying the inventory items, wherein the second production phase includes using outputs of the first inference engine over a period of time to identify gestures of the detected subjects carrying the inventory items; and
storing the identified gestures in a database.
2. The method of claim 1 , further comprising producing inventory events using the detected subjects, the identified inventory items and data representing the identified gestures and storing the inventory events as entries in the database.
3. The method of claim 2 , further including (i) determining a data set in the database, the data set including the inventory events for a particular inventory item in multiple locations in the area of real space, and (ii) displaying, on a user interface, a graphical construct indicating activity related to the particular inventory item in the multiple locations.
4. The method of claim 3 , wherein the activity related to the particular inventory item includes counts of the inventory events including the particular inventory item in the multiple locations in the period of time.
5. The method of claim 3 , wherein the activity related to the particular inventory item includes percentages of total inventory events, which include the particular inventory item, that have occurred in a particular location, of the multiple locations, in the period of time.
6. The method of claim 3 , wherein the activity related to the particular inventory item includes levels of the particular inventory item relative to a threshold count at the multiple locations.
7. The method of claim 3 , further including storing, with inventory events, a parameter indicating whether the particular inventory item is sold to a subject of the detected subjects.
8. The method of claim 3 , further including matching the inventory events for a selected inventory item in multiple locations to cells in a plurality of cells having coordinates in the area of real space.
9. The method of claim 8 , further including:
matching cells in the plurality of cells to inventory locations in inventory display structures in the area of real space; and
generating heat maps for inventory locations using the activity related to a selected inventory item in multiple locations.
10. The method of claim 2 , further including:
generating, for a particular inventory item, correlations to inventory events stored as entries in the database related to the particular inventory item; and
determining, for a selected subject, a temporally ordered sequence of inventory events including a particular inventory event.
11. The method of claim 10 , further including identifying an inventory item associated with an inventory event in the temporally ordered sequence of inventory events closest in time to the particular inventory event.
12. The method of claim 10 , further including identifying an inventory item associated with an inventory event in the temporally ordered sequence of inventory events closest in time to the particular inventory event that indicates that the particular inventory item is sold.
13. The method of claim 10 , further including filtering out inventory events, in the temporally ordered sequence of inventory events, including a parameter indicating inventory items as being sold to the selected subject.
14. The method of claim 10 , wherein the determining, for the selected subject, of the temporally ordered sequence of inventory events including the particular inventory event, operates without use of personal identifying biometric information associated with the selected subject.
15. The method of claim 2 , wherein the inventory events include a subject identifier identifying a detected subject, a gesture type of an identified gesture by the detected subject, an item identifier of an identified inventory item linked to the identified gesture by the detected subject, a location of the identified gesture represented by positions in three dimensions of the area of real space and a timestamp for the identified gesture.
16. The method of claim 1 , further including (i) implementing the first inference engine to classify a hand of a detected subject and to identify an inventory item held in the classified hand and (ii) implementing the second inference engine to identify a gesture of the classified hand.
17. A system for identifying gestures in an area of real space, the system comprising:
a plurality of sensors, sensors in the plurality of sensors producing respective sequences of frames of corresponding fields of view in the real space; and
a processing system coupled to the plurality of sensors and which detects subjects in the area of real space, the processing system having access to a database and including:
during a first production phase that implements a first inference engine that is pre-trained to operate in a first production mode, switching the first inference engine from a first training mode to the first production mode to identify inventory items carried by the detected subjects in the sequence of frames, wherein the first production phase includes using a sequence of frames produced by a corresponding sensor in the plurality of sensors to identify the inventory items carried by the detected subjects;
during a second production phase that implements a second inference engine that is pre-trained to operate in a second production mode, switching the second inference engine from a second training mode to the second production mode to identify gestures of the detected subjects carrying the inventory items, wherein the second production phase includes using outputs of the first inference engine over a period of time to identify gestures of the detected subjects carrying the inventory items; and
logic to store the identified gestures in the database.
18. The system of claim 17 , further including a plurality of first inference engines, the plurality of first inference engines including the first mentioned first inference engine, the plurality of first inference engines using the respective sequences of frames to identify inventory items held by detected subjects in the respective sequences of frames,
wherein the second inference engine uses outputs of more than one of the first inference engines over the period of time to identify gestures of the detected subjects.
19. The system of claim 17 , wherein the field of view of each sensor overlaps with the field of view of at least one other sensor in the plurality of sensors.
20. A non-transitory computer readable storage medium impressed with computer program instructions to track inventory events in an area of real space, the instructions when executed on a processor, implement a method comprising:
using a plurality of sensors to produce respective sequences of frames of corresponding fields of view in the area of real space;
detecting subjects in the area of real space;
during a first production phase that implements a first inference engine that is pre-trained to operate in a first production mode, switching the first inference engine from a first training mode to the first production mode to identify inventory items carried by the detected subjects in the sequence of frames, wherein the first production phase includes using a sequence of frames produced by a corresponding sensor in the plurality of sensors to identify the inventory items carried by the detected subjects;
during a second production phase that implements a second inference engine that is pre-trained to operate in a second production mode, switching the second inference engine from a second training mode to the second production mode to identify gestures of the detected subjects carrying the inventory items, wherein the second production phase includes using outputs of the first inference engine over a period of time to identify gestures of the detected subjects carrying the inventory items; and
storing the identified gestures in a database.