CONTEXTUALLY AWARE CUSTOMER ITEM ENTRY FOR AUTONOMOUS SHOPPING APPLICATIONS
A system and method for contextually aware customer item entry for autonomous shopping applications that includes an environmental sensing system distributed through a shopping environment that is configured to collect contextual data of customer activity in the environment; the environmental sensing system comprising at least a computer vision monitoring system, the computer vision monitoring system comprising a set of imaging devices distributed through the environment; a customer-managed item entry system that is movable through the environment by the customer and that is configured to collect item selection input; a virtual cart management module configured to manage a virtual cart with the item selection input and augmented at least in part by the contextual data, wherein the virtual cart is used in execution of an autonomous checkout process.
1 . A system for autonomous checkout comprising:
an environmental sensing system distributed through a shopping environment that is configured to collect contextual data of customer activity in the environment;
the environmental sensing system comprising at least a computer vision monitoring system, the computer vision monitoring system comprising a set of imaging devices distributed through the environment;
a customer-managed item entry system that is movable through the environment by the customer and that is configured to collect item selection input;
a virtual cart management module configured to manage a virtual cart with the item selection input and augmented at least in part by the contextual data, wherein the virtual cart is used in execution of an autonomous checkout process.
2 . The system of claim 1 , wherein the computer vision monitoring system is configured to generate an at least partial prediction of item selection by the customer; and wherein the at least partial prediction of a virtual cart is processed by the virtual cart management module to assess confidence in the virtual cart.
3 . The system of claim 1 , wherein the computer vision monitoring system comprises a customer tracking engine configured to track customer location; wherein the contextual data includes customer location relative to the environment; and wherein the contextual data is used in identification of items during management of the virtual cart.
4 . The system of claim 1 , wherein the environmental sensing system further comprises a smart shelving system.
5 . The system of claim 1 , wherein the item entry system is a customer application instance operable on a customer computing device.
6 . The system of claim 5 , wherein the customer computing device is a smart phone, where the customer application instance comprises a manual item entry user interface.
7 . The system of claim 5 , wherein the customer computing device is a pair of smart glasses that include a camera; and wherein the customer application is configured to automatically detect item selection events from image data collected from the camera of the smart glasses.
8 . The system of claim 1 , wherein the item entry system is a smart cart comprising at least one item detection sensor.
9 . The system of claim 8 , wherein the item detection sensor comprises a camera system; wherein the item selection input is the type of item selection event and item identification data from computer vision processing of image data from the camera system.
10 . The system of claim 9 , wherein the camera system comprises a first set of internal facing cameras directed inward at an item receptacle of the smart cart and a second set of cameras directed outward.
11 . The system of claim 8 , wherein the smart cart further comprises at least a second item detection sensor that is a digital scale.
12 . A method comprising:
at an environmental sensing system, collecting contextual data on customer activity in an environment;
managing a virtual cart through input from a customer-managed item entry system, the item entry system being movable through the environment, wherein managing the virtual cart comprises:
detecting an item selection event,
identifying an item involved in the selection event, and
updating a virtual cart with the item in accordance with the selection event; and
augmenting management of the virtual cart with the contextual state data.
13 . The method of claim 12 , wherein augmenting management of the virtual cart comprises augmenting the detection of an item selection event.
14 . The method of claim 12 , wherein augmenting management of the virtual cart comprises augmenting the identification of the item.
15 . The method of claim 14 , wherein the contextual data includes a set of candidate items in proximity to the customer; and wherein augmenting the identification of the item comprises prioritizing the set of candidate items when identifying the item.
16 . The method of claim 12 , wherein updating a virtual cart comprises assessing prediction confidence of the virtual cart; and wherein augmenting management of the virtual cart comprises augmenting assessment of the prediction confidence.
17 . The method of claim 12 , wherein the item entry system is a customer application instance operable on a customer computing device.
18 . The method of claim 17 , wherein detecting an item selection event and identifying an item involved in the selection event comprises receiving manual selection of an item through a user interface of the customer application instance.
19 . The method of claim 17 , wherein detecting an item selection event and identifying an item involved in the selection event comprises collecting image data from a camera of the item entry system and automatically detecting an item selection event and identifying the item from computer vision processing of the image data.
20 . The method of claim 12 , wherein the item entry system is a smart cart comprising at least one item detection sensor.