Granted Patent B2

US 12,591,847 · App. 17/963,751 · Granted Mar 31, 2026

Inventors: Yilun Chen (Dallas, TX); Ravi Kumar Dalal (Dallas, TX); Adam Cantor (Austin, TX); Monisha Elumalai (Dallas, TX); Benjamin R. Ellison (San Francisco, CA)

Assignee: Walmart Apollo, LLC

G06Q10/087G06T5/73G06T7/11G06V10/75G06V20/36G06V20/62G06V30/153G06T2207/20081G06T2207/20132

Abstract

Claims (55)

1 . An image based retail location confirmation system, comprising:

a machine learning model database storing a set of two or more machine learning models;

a deblur system communicatively coupled over a distributed communication network with the machine learning model database, wherein the deblur system is configured to receive a portion of a first image comprising a first presumed location label and a second presumed location label captured by an image capture device configured to capture images within a product storage facility, and apply at least a deblurring machine learning framework to the portion of the first image and generate a first deblurred label image comprising the first presumed location label and the second presumed location label;

a rectification system communicatively coupled over the distributed communication network with the machine learning model database, wherein the rectification system is configured to apply a rectification machine learning transform algorithm to justify the first deblurred label image according to a known shape of location labels to thereby generate a first rectified label image;

an optical character recognition (OCR) system communicatively coupled over the distributed communication network with the machine learning model database, wherein the OCR system is configured to:

apply a recognition machine learning model to the first rectified label image to estimate text of the first presumed location label and the second presumed location label; and

verify that identified patterns of the first presumed location label and the second presumed location label conform with a known location label pattern for location labels; and

a location estimation system configured to;

select one of the first presumed location label and the second presumed location label as a selected presumed location label based on which one of the first presumed location label and the second presumed location label is assigned a higher prioritized value for determining location; and

estimate a location within the product storage facility of the selected presumed location label as a function of the estimated text of the selected presumed location label relative to known text on known location labels positioned at respective different known locations within the product storage facility.

2 . The system of claim 1 , further comprising:

an image cropping system communicatively coupled over the distributed communication network with the machine learning model database, wherein the image cropping system is configured to apply a trained cropping machine learning model to identify within and extract from the first image the portion of the first image comprising the first presumed location label the second presumed location label.

3 . The system of claim 2 , further comprising:

a blur evaluation system configured to estimate a level of blur of the portion of the first image and enable the deblur system to generate the first deblurred label image when the estimated level of blur has a predefined relationship with a blur threshold.

4 . The system of claim 2 , further comprising:

a confidence evaluation system configured to apply at least one confidence rule to determine a confidence score of an accuracy of the estimated text of the first presumed location label and the second presumed location label; and

enable the location estimation system configured to estimate the location of the image capture device when the confidence score has a predefined relationship with a confidence threshold, and prevent the location estimation system from estimating locations of the image capture device when other confidence scores do not have the predefined relationship with the confidence threshold.

5 . The system of claim 4 , wherein the confidence evaluation system in applying the at least one confidence rule is configured to confirm that the estimated text complies with a predefined alphanumeric pattern of multiple alphanumeric characters.

6 . The system of claim 4 , further comprising:

a mobile task system comprising the image capture device, a task system, and a movement system communicatively coupled with the task system, wherein the task system is configured to implement code to control the movement system to control movement of the task system through the product storage facility;

wherein the image capture device is configured to capture the images as the task system moves through one or more portions of the product storage facility.

7 . The system of claim 4 , further comprising:

a portable user computing device comprising the image capture device, wherein the portable user computing device is configured to be transported by a user associated with the portable user computing device as the user moves through the product storage facility, and wherein the image capture device is configured to capture the images as the portable user computing device is transported through one or more portions of the product storage facility.

8 . The system of claim 4 , wherein the deblur system applying the deblurring machine learning framework is configured to apply a generative adversarial network (GAN) to the portion of the first image and generate the first deblurred label image comprising the first presumed location label and the second presumed location label.

9 . The system of claim 4 , further comprising:

a blur training database storing numerous training sets of actual images and numerous training sets of artificial images, wherein the numerous training sets of actual images each comprise: an actual image of one of the known location labels and at least one artificially blurred version of the one of the known location labels; and wherein the numerous training sets of artificial images each comprises: an artificially generated image of a representative location label based on known format, font and size of alphanumeric characters included on the known location labels, and at least one artificially blurred version of the artificially generated image; and

a machine learning training system communicatively coupled over the distributed communication network with the machine learning model database and the blur training database, wherein the machine learning training system is configured to repeatedly train over time the deblurring machine learning framework utilizing the numerous training sets of actual images and the numerous training sets of artificial images.

10 . A method of confirming locations within a product storage facility, comprising:

storing, in a machine learning model database, a set of two or more machine learning models;

receiving images captured by an image capture device configured to capture the images within the product storage facility;

receiving a portion of a first image, of the images, wherein the portion of the first image comprises a first presumed location label and a second presumed location label captured by the image capture device;

applying a deblurring machine learning framework to the portion of the first image and generating a first deblurred label image comprising the first presumed location label and the second presumed location label;

applying a rectification machine learning transform algorithm to justify the first deblurred label image according to a known shape of location labels and thereby generating a first rectified label image;

applying a recognition machine learning model to the first rectified label image to:

estimate text of the first presumed location label and the second presumed location label; and

verify that identified patterns of the first presumed location label and the second presumed location label conform with a known location label pattern for location labels;

selecting one of the first presumed location label and the second presumed location label as a selected presumed location label based on which one of the first presumed location label and the second presumed location label is assigned a higher prioritized value for determining location; and

estimating a location within the product storage facility of the selected presumed location label as a function of the estimated text of the selected presumed location label relative to known text on known location labels position at respective different known locations within the product storage facility.

11 . The method of claim 10 , further comprising:

applying a trained cropping machine learning model to identify within and extract from the first image the portion of the first image comprising the first presumed location label and the second presumed location label.

12 . The method of claim 11 , further comprising:

estimating, by a blur evaluation system, a level of blur of the portion of the first image and enabling the generating of the first deblurred label image when the estimated level of blur has a predefined relationship with a blur threshold.

13 . The method of claim 11 , further comprising:

applying at least one confidence rule and determining a confidence score of an accuracy of the estimated text of the first presumed location label and the second presumed location label; and

enabling the estimating of the location of the image capture device when the confidence score has a predefined relationship with a confidence threshold, and preventing the estimation of locations of the image capture device when other confidence scores do not have the predefined relationship with the confidence threshold.

14 . The method of claim 13 , wherein the applying the at least one confidence rule comprises confirming that the estimated text complies with a predefined alphanumeric pattern of multiple alphanumeric characters.

15 . The method of claim 13 , further comprising:

controlling a movement system of a mobile task system in controlling movement of the mobile task system through the product storage facility, wherein the mobile task system comprises the image capture device; and

controlling the image capture device to capture the images as the mobile task system moves through one or more portions of the product storage facility.

16 . The method of claim 13 , further comprising:

controlling the image capture device of a portable user computing device to capture the images as the portable user computing device is transported through one or more portions of the product storage facility by a user associated with the portable user computing device as the user moves through the product storage facility.

17 . The method of claim 13 , wherein the applying the deblurring machine learning framework comprises applying a generative adversarial network (GAN) to the portion of the first image and generating the first deblurred label image comprising the first presumed location label and the second presumed location label.

18 . The method of claim 13 , further comprising:

generating and storing in a blur training database storing numerous training sets of actual images and numerous training sets of artificial images, wherein the numerous training sets of actual images each comprise: an actual image of one of the known location labels and at least one artificially blurred version of the one of the known location labels; and wherein the numerous training sets of artificial images each comprises: an artificially generated image of a representative location label based on known format, font and size of alphanumeric characters included on the known location labels, and at least one artificially blurred version of the artificially generated image; and

repeatedly training over time the deblurring machine learning framework utilizing the numerous training sets of actual images and the numerous training sets of artificial images.

Assignments (2)

Continuity (1)

References Cited (128)