Smart cart issue prediction with remedial workflow

1 . A method, performed by a computer system comprising a processor and a non-transitory computer-readable medium, comprising:

obtaining sensor data from one or more sensors of a smart cart, wherein obtaining the sensor data comprises:

obtaining wheel rotation data from one or more wheel sensors coupled to wheels of the smart cart, the wheel rotation data indicating rotation of the one or more wheels, and

obtaining inertial measurement unit (IMU) data from an IMU sensor coupled to the smart cart;

applying a cart issue prediction model trained as a machine-learning model to input the sensor data comprising the wheel rotation data and the IMU data and to identify one or more predicted issues with operation of the smart cart, wherein the cart issue prediction model is trained according to a process comprising:

obtaining historical sensor data for a fleet of smart carts operated in an in-store environment, wherein the historical sensor data comprises wheel rotation data captured by wheel sensors coupled to the fleet of smart carts and IMU data captured by IMU sensors coupled to the fleet of smart carts, and wherein the historical sensor data for each smart cart in the fleet is associated with a label indicating any anomalies in the operation of the smart cart, and

training the machine-learning model with the historical sensor data and the labels for the fleet of smart carts to identify anomalies in operation of a given smart cart based on the sensor data of the given smart cart; and

identifying the one or more predicted issues with the operation of the smart cart by the cart issue prediction model trained as the machine learning model;

receiving user input, via a client device, confirming that the smart cart is inoperable based on the one or more predicted issues with the operation of the smart cart;

responsive to the smart cart being inoperable, generating a command signal to power down the smart cart, and

transmitting the command signal to the smart cart.

2 . The method of claim 1 , wherein obtaining the sensor data comprises obtaining one or more types of sensor data including one or more of: item scan data from one or more item scanners in view of one or more baskets of the smart cart, the item scan data indicating one or more items corresponding to item identifies scanned by the item scanners; load data from one or more load sensors coupled to the one or more baskets of the smart cart, the load data indicating load on each basket over time; or image data captured from one or more cameras in view of the one or more baskets of the smart cart.

3 . The method of claim 2 , wherein training the cart issue prediction model further comprises: building a normal distribution for each type of sensor data based on the sensor data pertaining to that type of sensor data, wherein the cart issue prediction model is trained to identify cart issues based on sensor data that is anomalous compared to the normal distributions.

4 . The method of claim 1 further comprising: obtaining user feedback from a user of the smart cart relating to operation of the smart cart in the in-store environment, wherein applying the cart issue prediction model comprises applying the cart issue prediction model further to the user feedback to identify the one or more predicted issues with operation of the smart cart.

5 . The method of claim 1 , wherein training the cart issue prediction model further comprises: obtaining service records for the fleet of smart carts; identifying one or more issues in operation of the one or more smart carts in the fleet of smart carts; and generating training examples with the historical sensor data and the one or more issues identified from the service records; wherein training the cart issue prediction model comprises training the cart issue prediction model in a supervised manner with the training examples.

6 . The method of claim 1 , wherein applying the cart issue prediction model to the sensor data comprises applying a plurality of submodels to the sensor data to identify the one or more cart issues, wherein each submodel is trained to identify one type of cart issue.

7 . The method of claim 1 , wherein applying the cart issue prediction model comprises: applying one submodel to the IMU data to identify anomalous frequencies in the IMU data compared to IMU data from normal cart operation of the fleet of smart carts; and identifying one or more predicted issues with the operation of the smart cart based on the identified anomalous frequencies.

8 . The method of claim 1 , wherein obtaining the sensor data comprises: obtaining item scan data from one or more item scanners in view of one or more baskets of the smart cart, the item scan data indicating one or more items corresponding to item identifiers scanned by the item scanners, obtaining load data from one or more load sensors coupled to a storage area of the smart cart, the load data indicating a load on the storage area from the one or more items, and image data captured from one or more cameras depicting the one or more items in the storage area of the smart cart; and wherein applying the cart issue prediction model comprises: identifying an item placed in one basket of the smart cart based on the item scan data or the image data; identifying load of the item from the load data; retrieving weight information on the item placed in the one basket from a database storing characteristics of items; comparing the load of the item to the weight information on the item; and responsive to a difference between the load of the item and the weight information of the item being above a threshold, identifying a predicted issue with the load sensor.

9 . The method of claim 1 , wherein applying the cart issue prediction model comprises: applying one submodel to the wheel rotation data and the IMU data to identify a predicted issue with miscalibration of either the one or more wheel sensors or the IMU sensor.

10 . The method of claim 1 , further comprising: identifying a calibration error in a sensor of the smart cart; generating a command signal to calibrate the sensor of the smart cart; and transmitting the command signal to the smart cart for calibration of the sensor.

11 . The method of claim 1 , further comprising: identifying one or more remedial tasks instructing a user of the smart cart to perform; and transmitting the one or more remedial tasks to a client device of the smart cart for presentation to the user.

12 . The method of claim 11 , wherein identifying the one or more remedial tasks to instruct the user of the smart cart to perform comprises one or more of: identifying a first remedial task instructing the user to inspect the smart cart to verify presence of the one or more predicted issues; identifying a second remedial task instructing the user to modify operation of the smart cart; and identifying a third remedial task instructing the user to relocate the smart cart to a service site in the in-store environment.

13 . The method of claim 1 , further comprising: generating an issue report listing out the one or more predicted issues; and transmitting a service appointment scheduling request with the issue report.

14 . The method of claim 1 , further comprising: receiving feedback from a user validating or invalidating the one or more predicted issues; generating additional training examples based on the feedback; and fine tuning the machine-learning model based on the additional training examples.

15 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor, cause the computer processor to perform operations comprising:

obtaining sensor data from one or more sensors of a smart cart, wherein obtaining the sensor data comprises: obtaining wheel rotation data from one or more wheel sensors coupled to wheels of the smart cart, the wheel rotation data indicating rotation of the one or more wheels, and obtaining inertial measurement unit (IMU) data from an IMU sensor coupled to the smart cart;

applying a cart issue prediction model trained as a machine-learning model to input the sensor data comprising the wheel rotation data and the IMU data and to identify one or more predicted issues with operation of the smart cart, wherein the cart issue prediction model is trained according to a process comprising: obtaining historical sensor data for a fleet of smart carts operated in an in-store environment, wherein the historical sensor data comprises wheel rotation data captured by wheel sensors coupled to the fleet of smart carts and IMU data captured by IMU sensors coupled to the fleet of smart carts, and wherein the historical sensor data for each smart cart in the fleet is associated with a label indicating any anomalies in the operation of the smart cart, and

training the machine-learning model with the historical sensor data and the labels for the fleet of smart carts to identify anomalies in operation of a given smart cart based on the sensor data of the given smart cart; and

identifying the one or more predicted issues with the operation of the smart cart by the cart issue prediction model trained as the machine learning model;

receiving user input, via a client device, confirming that the smart cart is inoperable based on the one or more predicted issues with the operation of the smart cart;

responsive to the smart cart being inoperable, generating a command signal to power down the smart cart; and

transmitting the command signal to the smart cart.

16 . The non-transitory computer-readable storage medium of claim 15 , wherein applying the cart issue prediction model comprises: applying one submodel to the IMU data to identify anomalous frequencies in the IMU data compared to IMU data from normal cart operation of the fleet of smart carts; and identifying one or more predicted issues with the operation of the smart cart based on the identified anomalous frequencies.

17 . The non-transitory computer-readable storage medium of claim 15 , wherein obtaining the sensor data comprises: obtaining item scan data from one or more item scanners in view of one or more baskets of the smart cart, the item scan data indicating one or more items corresponding to item identifiers scanned by the item scanners, obtaining load data from one or more load sensors coupled to a storage area of the smart cart, the load data indicating a load on the storage area from the one or more items, and image data captured from one or more cameras depicting the one or more items in the storage area of the smart cart; and wherein applying the cart issue prediction model comprises: identifying an item placed in one basket of the smart cart based on the item scan data or the image data; identifying load of the item from the load data; retrieving weight information on the item placed in the one basket from a database storing characteristics of items; comparing the load of the item to the weight information on the item; and responsive to a difference between the load of the item and the weight information of the item being above a threshold, identifying a predicted issue with the load sensor.

18 . The non-transitory computer-readable storage medium of claim 15 , wherein applying the cart issue prediction model comprises: applying one submodel to the wheel rotation data and the IMU data to identify a predicted issue with miscalibration of either the one or more wheel sensors or the JIU sensor.

19 . A system comprising: a computer processor; and a non-transitory computer-readable storage medium storing instructions that, when executed by the computer processor, cause the computer processor to perform operations comprising:

obtaining sensor data from one or more sensors of a smart cart, wherein obtaining the sensor data comprises: obtaining wheel rotation data from one or more wheel sensors coupled to wheels of the smart cart, the wheel rotation data indicating rotation of the one or more wheels, and obtaining inertial measurement unit (IMU) data from an IMU sensor coupled to the smart cart;

applying a cart issue prediction model trained as a machine-learning model to input the sensor data comprising the wheel rotation data and the IMU data and to identify one or more predicted issues with operation of the smart cart, wherein the cart issue prediction model is trained according to a process comprising: obtaining historical sensor data for a fleet of smart carts operated in an in-store environment, wherein the historical sensor data comprises wheel rotation data captured by wheel sensors coupled to the fleet of smart carts and IMU data captured by IMU sensors coupled to the fleet of smart carts, and wherein the historical sensor data for each smart cart in the fleet is associated with a label indicating any anomalies in the operation of the smart cart, and training the machine-learning model with the historical sensor data and the labels for the fleet of smart carts to identify anomalies in operation of a given smart cart based on the sensor data of the given smart cart;

identifying the one or more predicted issues with the operation of the smart cart by the cart issue prediction model trained as the machine learning model;

receiving user input, via a client device, confirming that the smart cart is inoperable based on the one or more predicted issues with the operation of the smart cart;

responsive to the smart cart being inoperable, generating a command signal to power down the smart cart; and

transmitting the command signal to the smart cart.