Determine an item and user action in a materials handling facility
Described is a system and method for presenting event information to a user and, if necessary, obtaining confirmation of different aspects (user, item, action) of the event. In some implementations, an event includes a user, an action, and an item. For example, an event may include a user picking an item from an inventory location, a user placing an item into a tote associated with the user, etc. if the aspects of the event cannot be determined with a high enough degree of confidence, a user interface may be generated and sent to the user requesting confirmation of one or more of the aspects of the event.
1. A system, comprising:
a plurality of cameras positioned within a materials handling facility, each camera of the plurality of cameras configured to generate image data of objects located within the materials handling facility;
one or more processors; and
a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to at least:
detect an occurrence of an event performed by a user within a materials handling facility;
receive, from at least one of the plurality of cameras, an image of the user performing the event, wherein the image includes at least a portion of a representation of the user and at least a portion of a representation of an item;
process the image to determine a user identity of the user represented in the image;
determine a user identification confidence score representative of a confidence that the user identity of the user is correct;
determine an action determination confidence score representative of a confidence that a determined action performed by the user as part of the event is correct;
process the image to determine an item identity of the item represented in the image;
determine an item determination confidence score representative of a confidence that the item identity of the item is correct;
determine that the item determination confidence score does not exceed an item confidence threshold; and
send data for use in presenting to the user a user interface identifying the user identity, the action and the item identity, wherein:
the presentation on the user interface includes a request that the user confirm the item identity of the item involved in the event is correct; and
the action is at least one of an item pick from an inventory location, an item place at the inventory location, an item pick from a tote, or an item place into the tote.
2. The system of claim 1 , wherein the program instructions further cause the one or more processors to at least:
send data for use in presenting to the user alternative item information representative of an alternative item that is different than the item, wherein the alternative item information may be selected by the user to confirm that the alternative item was involved in the event; and
receive a selection of the alternative item information.
3. The system of claim 1 , wherein the program instructions further cause the one or more processors to at least:
determine that the action determination confidence score does not exceed an action confidence threshold; and
wherein the data sent for use in presenting to the user the user interface further includes a second request that the user confirm the action performed by the user.
4. A system, comprising:
a plurality of cameras positioned within a materials handling facility, each camera of the plurality of cameras configured to generate image data of objects located within the materials handling facility;
one or more processors; and
a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to at least:
determine that a user has performed an action;
receive, from at least one camera of the plurality of cameras, image data of the user performing the action;
process the image data to determine a user identity of the user and an item identity of an item;
determine:
a user identification confidence score representative of a confidence that the user identity of the user is correct;
an action determination confidence score representative of a confidence that a determined action performed by the user is correct; and
an item determination confidence score representative of a confidence that the item identity of the item involved in the action is correct;
determine that at least one of the action determination confidence score or the item determination confidence score does not exceed a threshold; and
send for presentation to the user:
item information that includes a representation of the item;
action information representative of the action performed by the user; and
a request that the user confirm at least one of: the action performed by the user; the item identity; or that the user did not perform the action, wherein the request is based at least in part on one or more of the determination that at least one of the action determination confidence score or the item determination confidence score do not exceed a threshold.
5. The system of claim 4 , wherein the item information, the action information, and the request are sent to a device associated with the user.
6. The system of claim 4 , wherein the program instructions further cause the one or more processors to at least:
send a notification to a device to alert the user about the request.
7. The system of claim 6 , wherein the notification is sent to a different device than the device to which the item information, the action information or the request is sent.
8. The system of claim 6 , wherein the notification causes a device to vibrate, illuminate a display, or emit an audible sound.
9. The system of claim 4 , wherein the one or more processors are in network communication with a device accessible to the user.
10. The system of claim 9 , wherein the device accessible to the user is at least one of: a portable device associated with the user, a device coupled to a tote associated with the user, a user's device, a display within a materials handling facility, or a projector within the materials handling facility.
11. The system of claim 4 , wherein:
the program instructions that cause the one or more processors to process the image data further include instructions that cause the one or more processors to at least:
process the image data of the user using at least one of facial recognition, pattern matching, or object matching; and
the program instructions further cause the one or more processors to at least:
for each of a plurality of stored user image information, determine a correlation score between the processed image and the stored user image information; and
determine a user identification confidence score based at least in part on a ratio between at least two of the plurality of correlation scores.
12. The system of claim 11 , wherein the program instructions further cause the one or more processors to at least:
detect a user identifier associated with the user, wherein the user identifier is at least one of a visual tag, an active tag, a user name, or a biometric of the user; and
wherein the user identifier confidence score is based at least in part on the detected user identifier.
13. The system of claim 4 , wherein the action determination confidence score is based at least in part on: a detected change in a weight at an inventory location, or a detected change in a weight in a tote.