Method and medium for shopping in a physical store

1. A computer implemented method for assisting in shopping at a physical store, the method including:

after a customer device is linked to a cart system, activating a camera of the cart system that is mechanically coupled to a shopping cart, the camera configured to capture image data of an inside of the shopping cart;

receiving the image data;

processing the image data to detect one or more events and, for each event, determining an event type, the event type being an item addition event involving the addition of an item to the shopping cart or an item removal event involving the removal of the item from the shopping cart;

responsive to detecting an event:

processing the image data to identify an item associated with the event;

generating cart information based on the identified item, the cart information including a record of items currently in the shopping cart;

communicating the cart information with respect to the identified item to the customer device;

detecting the shopping cart in an exit area; and

responsive to detecting the shopping cart in the exit area, automatically processing a payment based on the invoice; and

responsive to processing the payment, deactivating the camera.

2. The method of claim 1 , wherein generating the cart information includes:

retrieving item information in respect of the identified item, the item information including at least a price of the identified item; and

generating an invoice based on the prices of items recorded as being currently in the shopping cart.

3. The method of claim 2 , wherein communicating the cart information to the customer device includes communicating the invoice and item information to the customer device.

4. The method of claim 1 , further including

authenticating the customer device.

5. The method of claim 1 , wherein processing the image data includes:

receiving at least a portion of the image data corresponding to the detected event;

generating an image mask for each image in the received portion of the image data, wherein the image mask highlights an item in the image and masks the remaining image; and

identifying the highlighted item.

6. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processing units, cause a machine to perform operations comprising:

activating a camera of a cart system mechanically coupled to a shopping cart at the physical store, the cart system configured to capture footage of the inside of the shopping cart during a shopping session;

receiving image data;

processing the image data to detect one or more events and, for each event, determining the event type, the event type being an item addition event involving the addition of an item to the shopping cart or an item removal event involving the removal of an item from the shopping cart;

processing the image data to identify an item associated with an event, responsive to detecting the event;

generating cart information based on the identified item, the cart information including a record of items currently in the shopping cart;

communicating the cart information with respect to the identified item to a customer device;

detecting the shopping cart in an exit area; and

responsive to detecting the shopping cart in the exit area, automatically processing a payment based on the invoice; and

responsive to processing the payment, deactivating the camera.

7. The non-transitory computer-readable medium of claim 6 , further comprising instructions that, when executed by one or more computer processing units, further cause the machine to perform operations comprising:

retrieving item information in respect of the identified item, the item information including at least a price of the identified item; and

generating an invoice based on the prices of items recorded as being currently in the shopping cart.

8. The non-transitory computer-readable medium of claim 6 , further comprising instructions that, when executed by one or more computer processing units, further cause the machine to perform operations comprising:

in response to detecting the shopping cart in the exit area,

communicating exit detection information to a backend system.

9. The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by one or more computer processing units, further cause the machine to perform operations comprising:

initiating the shopping session and linking the customer device with the cart system.

10. The non-transitory computer-readable medium of claim 9 , further comprising instructions that, when executed by one or more computer processing units, further cause the machine to perform operations comprising:

sending an activation signal to the cart system to activate the camera after the customer device is linked to the cart system; and

sending a deactivation signal to the cart system to deactivate the camera after payment is processed.

11. The non-transitory computer-readable medium of claim 6 , further comprising instructions that, when executed by one or more computer processing units, further cause the machine to perform operations comprising:

generating an image mask on a received image, the image mask highlighting the detected item and masking the remaining image, and an item recognition neural network model configured to identify the detected item highlighted in the image mask.