Customer assistance at self checkouts using computer vision
Techniques relating to using machine learning (ML) with a point of sale (POS) system. These techniques include identifying one or more images, captured at a POS system, of one or more items for purchase. The techniques further include predicting a checkout issue relating to the one or more items for purchase, including determining the checkout issue using a trained ML model, based on the one or more images. The techniques further include generating one or more instructions for a purchaser of the one or more items, based on the predicted checkout issue, and presenting the one or more instructions at a user interface of the POS system.
1 . A method comprising:
identifying one or more images, captured at a point of sale (POS) system, of one or more items for purchase;
receiving discrepancy weight data, captured at the POS system, associated with the one or more items for purchase;
predicting a checkout issue relating to the one or more items for purchase, comprising:
determining the checkout issue using a trained machine learning (ML) model, based on the one or more images, comprising:
generating a feature vector for discrepancy weight data and each trained one or more images of the items for purchase;
compressing the feature vector for the discrepancy weight data each of the trained one or more images into a vector data stream;
streaming the vector data stream into the computer vision ML model;
generating a first output at a computer vision ML model trained to recognize items for purchase, based on streaming the vector data stream into the computer vision ML model;
predicting the checkout issue by streaming the first output to the trained ML model, wherein the trained ML model is different from the computer vision ML model;
generating one or more instructions for a purchaser of the one or more items, based on the predicted checkout issue; and
generating a user interface at the POS system, wherein the user interface includes graphical elements comprising the one or more instructions.
2 . The method of claim 1 , wherein the generating one or more instructions for the purchaser of the one or more items, based on the predicted checkout issue, comprises:
determining the instructions using a natural language processing (NLP) ML model, based on providing the predicted checkout issue to the NLP ML model.
3 . The method of claim 2 , wherein the one or more instructions comprise instructions to improve accuracy of identifying the one or more items for purchase using the computer vision ML model.
4 . The method of claim 1 , wherein determining the checkout issue using the trained ML model, based on the one or more images, comprises:
predicting the checkout issue by providing the one or more images to the trained ML model, wherein the trained ML model outputs the predicted checkout issue.
5 . The method of claim 4 , wherein the generating one or more instructions for the purchaser of the one or more items, based on the predicted checkout issue, comprises:
determining the instructions using a natural language processing (NLP) ML model, based on providing the predicted checkout issue to the NLP ML model.
6 . The method of claim 5 , wherein the one or more instructions comprise instructions to improve accuracy of identifying the one or more items for purchase using a computer vision ML model.
7 . The method of claim 1 , wherein predicting the checkout issue relating to the one or more items for purchase is based on weight data captured using a scale, in addition to the one or more images.
8 . The method of claim 1 , wherein the computer vision ML model and the trained ML model comprise neural networks, deep neural networks, or a support vector machine.
9 . The method of claim 1 , wherein the first output comprises confidence scores relating to identified items.
10 . The method of claim 1 , wherein the one or more instructions for a purchaser of the one or more items, are based on confidence scores provided by the trained ML model in addition to the predicted checkout issue.
11 . A non-transitory computer program product comprising:
one or more non-transitory computer readable media containing, in any combination, computer program code that, when executed by operation of any combination of one or more processors, performs operations comprising:
identifying one or more images, captured at a point of sale (POS) system, of one or more items for purchase;
receiving discrepancy weight data, captured at the POS system, associated with the one or more items for purchase;
predicting a checkout issue relating to the one or more items for purchase, comprising:
determining the checkout issue using a trained machine learning (ML) model, based on the one or more images, comprising:
generating a feature vector for discrepancy weight data and each trained one or more images of the items for purchase;
compressing the feature vector for the discrepancy weight data each of the trained one or more images into a vector data stream;
streaming the vector data stream into the computer vision ML model;
generating a first output at a computer vision ML model trained to recognize items for purchase, based on streaming the vector data stream into the computer vision ML model;
predicting the checkout issue by streaming the first output to the trained ML model, wherein the trained ML model is different from the computer vision ML model;
generating one or more instructions for a purchaser of the one or more items, based on the predicted checkout issue; and
generating a user interface at the POS system, wherein the user interface includes graphical elements comprising the one or more instructions.
12 . The non-transitory computer program product of claim 11 , wherein the one or more instructions comprise instructions to improve accuracy of identifying the one or more items for purchase using the computer vision ML model.
13 . The non-transitory computer program product of claim 11 , wherein determining the checkout issue using the trained ML model, based on the one or more images, comprises:
predicting the checkout issue by providing the one or more images to the trained ML model, wherein the trained ML model outputs the predicted checkout issue.
14 . The non-transitory computer program product of claim 13 , wherein the generating one or more instructions for the purchaser of the one or more items, based on the predicted checkout issue, comprises:
determining the instructions using a natural language processing (NLP) ML model, based on providing the predicted checkout issue to the NLP ML model,
wherein the one or more instructions comprise instructions to improve accuracy of identifying the one or more items for purchase using a computer vision ML model.
15 . The non-transitory computer program product of claim 11 , wherein the computer vision ML model and the trained ML model comprise neural networks, deep neural networks, or a support vector machine.
16 . A system, comprising:
one or more processors; and
one or more memories storing a program, which, when executed on any combination of the one or more processors, performs operations, the operations comprising:
identifying one or more images, captured at a point of sale (POS) system, of one or more items for purchase;
receiving discrepancy weight data, captured at the POS system, associated with the one or more items for purchase;
predicting a checkout issue relating to the one or more items for purchase, comprising:
determining the checkout issue using a trained machine learning (ML) model, based on the one or more images, comprising:
generating a feature vector for discrepancy weight data and each trained one or more images of the items for purchase;
compressing the feature vector for the discrepancy weight data each of the trained one or more images into a vector data stream;
streaming the vector data stream into the computer vision ML model;
generating a first output at a computer vision ML model trained to recognize items for purchase, based on streaming the vector data stream into the computer vision ML model;
predicting the checkout issue by streaming the first output to the trained ML model, wherein the trained ML model is different from the computer vision ML model;
generating one or more instructions for a purchaser of the one or more items, based on the predicted checkout issue; and
generating a user interface at the POS system, wherein the user interface includes graphical elements comprising the one or more instructions.
17 . The system of claim 16 wherein the one or more instructions comprise instructions to improve accuracy of identifying the one or more items for purchase using the computer vision ML model.
18 . The system of claim 16 , wherein determining the checkout issue using the trained ML model, based on the one or more images, comprises:
predicting the checkout issue by providing the one or more images to the trained ML model, wherein the trained ML model outputs the predicted checkout issue.
19 . The system of claim 18 , wherein the generating one or more instructions for the purchaser of the one or more items, based on the predicted checkout issue, comprises:
determining the instructions using a natural language processing (NLP) ML model, based on providing the predicted checkout issue to the NLP ML model,
wherein the one or more instructions comprise instructions to improve accuracy of identifying the one or more items for purchase using a computer vision ML model.
20 . The system of claim 16 , wherein the computer vision ML model and the trained ML model comprise neural networks, deep neural networks, or a support vector machine.