IP Library Granted Patent US 10,984,239
Granted Patent B2
US 10,984,239 · App. 16/144,337 · Granted Apr 20, 2021

Context-aided machine vision

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,984,239
App. No.
16/144,337
Granted
Apr 20, 2021
Kind
B2
Abstract

Various embodiments herein each include at least one of systems, methods, software, and data structures for context-aided machine vision. For example, one method embodiment includes identifying a customer in a shopping area and maintaining an item bin in a computing system of data identifying items the customer has picked up for purchase. This method further includes receiving an image of the customer holding an item and performing item identification processing on the image to identify the item the customer is holding. The item identification processing may be performed based in part on a stored shopping history of the customer indicating items the customer is more likely to purchase. The identified item is then added to the item bin of the customer.

Claims (45)

1. A method comprising:

identifying a customer in a shopping area;

maintaining an item bin in a computing system of data identifying items the customer has picked up for purchase;

receiving an image of the customer holding an item;

performing item identification processing on the image to identify the item the customer is holding, the item identification processing based at least in part on a stored shopping history of the customer indicating items the customer is more likely to purchase the item identification processing including applying a multi-dimensional model with data representing a customer likelihood to purchase certain items and to not purchase other items; and

adding an item identified by the item identification processing to the item bin of the customer.

2. The method of claim 1 , wherein the model further includes location, day of the week, and time of day factors indicating a customer likelihood to purchase or not to purchase certain items.

3. The method of claim 1 , wherein the model is utilized by the item identification processing to obtain a weight with regard to each of one or more item possibilities identified by the image processing, wherein the weight with regard to item possibility is a factor for the item identification processing to determine an identity of the item the customer is holding.

4. The method of claim 1 , wherein a model is generated for each of a plurality of customers by processing customer shopping history data stored in a database with at least one machine learning algorithm.

5. The method of claim 4 , wherein the machine learning algorithm is a convolutional neural network algorithm.

6. The method of claim 1 , wherein identifying a customer in a shopping area includes:

capturing an image of at least a portion of a shopping area; and

performing a person recognition function on the image to obtain a customer identity.

7. The method of claim 1 , wherein receiving the image of the customer holding the item includes receiving a cropped image that has been cropped to focus item identification processing on an image portion of a larger image that includes the item.

8. A method comprising:

receiving an output from an image processing function that operates to identify an item possessed by an identified individual in a processed image, the output includes an identifier of the individual, identifiers of each of a plurality of items, and a probability of an item of each respective item identifier matching the item possessed by the individual;

processing each item identifier of the received output to determine a respective weight to be combined with the probability of the item matching the item possessed by the individual, each respective weight determined based on at least one non-image data models, the processing including:

identifying at least one model based on the identifier of the individual, the model generated from data representative of historic item selection behavior of the identified individual; and

applying each of the at least one identified models with regard to each item identifier of the received output to determine the respective weights;

combining the determined weights with respective probabilities to declare an item identifier as the most likely match to the item possessed by the individual; and

outputting the item identifier as the item possessed by the individual to an item bin data structure.

9. The method of claim 8 , wherein the data representative of historic item selection behavior of the identified individual includes a data representation of items purchased by the identified customer.

10. The method of claim 9 , wherein the data representation of items purchased by the identified customer further includes at least one of day of the week and time data of when a respective product has been purchased.

11. The method of claim 8 , wherein:

the output of the image processing function further includes data identifying an area represented in the processed image;

the at least one non-image data model includes an item location model including data associating item identifiers to respective locations within a facility; and

the processing of each item identifier of the received output to determine a respective weight to be combined with the probability of the item matching the item possessed by the individual includes applying the item location model with regard to each item identifier of the received output and the data identifying the area represented in the processed image to determine the respective weights.

12. The method of claim 8 , wherein the probabilities and weights are scoring elements that are summed by the combining.

13. A system comprising:

a computer processor;

a memory device; and

instructions stored on the memory device that are executable by the computer processor to perform data processing activities comprising:

receiving an output from an image processing function that operates to identify an item possessed by an identified individual in a processed image, the output includes an identifier of the individual, identifiers of each of a plurality of items, and a probability of an item of each respective item identifier matching the item possessed by the individual;

processing each item identifier of the received output to determine a respective weight to be combined with the probability of the item matching the item possessed by the individual, each respective weight determined based on at least one non-image data models, the processing including:

identifying at least one model based on the identifier of the individual, the model generated from data representative of historic item selection behavior of the identified individual; and

applying each of the at least one identified models with regard to each item identifier of the received output to determine the respective weights;

combining the determined weights with respective probabilities to declare an item identifier as the most likely match to the item possessed by the individual; and

outputting the item identifier as the item possessed by the individual to an item bin data structure.

14. The system of claim 13 , wherein the data representative of historic item selection behavior of the identified individual includes a data representation of items purchased by the identified customer.

15. The system of claim 14 , wherein the data representation of items purchased by the identified customer further includes at least one of day of the week and time data of when a respective product has been purchased.

16. The system of claim 13 , wherein:

the output of the image processing function further includes data identifying an area represented in the processed image;

the at least one non-image data model includes an item location model including data associating item identifiers to respective locations within a facility; and

the processing of each item identifier of the received output to determine a respective weight to be combined with the probability of the item matching the item possessed by the individual includes applying the item location model with regard to each item identifier of the received output and the data identifying the area represented in the processed image to determine the respective weights.

17. The system of claim 13 , wherein the probabilities and weights are scoring elements that are summed by the combining.