Granted Patent B2

US 12,412,149 · App. 18/161,788 · Granted Sep 9, 2025

Inventors: Raghava Balusu (Achanta, IN); Siddhartha Chakraborty (Kolkata, IN); Ashlin Ghosh (Ernakulam, IN); Avinash M. Jade (Bangalore, IN); Lingfeng Zhang (Dallas, TX); Amit Jhunjhunwala (Bangalore, IN)

Assignee: Walmart Apollo, LLC

G06Q10/087G06V10/762

Abstract

Claims (46)

1. A system for processing captured images of objects at a product storage facility, the system comprising:

a memory configured to store a plurality of images, the plurality of images comprising first images and second images, wherein each of the first images contain items not detected by a trained machine learning model as being associated with a recognized product identifier, and wherein each of the second images contain items detected by the trained machine learning model as being associated with multiple recognized product identifiers; and

a control circuit executing the trained machine learning model configured to:

allocate each of the first images into one of a plurality of datasets based on areas in the product storage facility the first images were captured;

for each dataset of the plurality of datasets, cluster each image in the dataset into one of a plurality of groups based on a degree of resemblance of items depicted in the image relative to those items depicted in other images in the dataset;

select a sample from at least one of the plurality of groups;

cluster each of the second images into one of a dominant product identifier group and a non-dominant product identifier group;

select a sample from the dominant product identifier group;

select a sample from the non-dominant product identifier group; and

output at least one of the selected sample from the at least one of the plurality of groups, the selected sample from the dominant product identifier group, and the selected sample from the non-dominant product identifier group to be used to retrain the trained machine learning model.

2. The system of claim 1 , wherein the plurality of groups comprise a homogenous group, a heterogeneous group, a low similarity group, and an individual group.

3. The system of claim 2 , wherein the homogenous group includes images where the degree of resemblance of items depicted in each image is greater than 70%.

4. The system of claim 2 , wherein the heterogeneous group includes images where the degree of resemblance of items depicted in each image is at least 30% but no more than 70%.

5. The system of claim 2 , wherein the low similarity group includes images where the degree of resemblance of items depicted in each image is less than 30%.

6. The system of claim 2 , wherein the individual group includes images that do not have resemblance with any other image in the dataset.

7. The system of claim 2 , wherein the control circuit executing the trained machine learning model is further configured to:

process each image included in the low similarity group by determining the degree of resemblance of items depicted in the image, and clustering the image into one of the homogenous group, the heterogeneous group, the low similarity group, or the individual group in accordance with the determined degree of resemblance of the items; and

repeat processing of subsequent images included in the low similarity group until each of the subsequent images is clustered into one of the homogenous group, the heterogeneous group, or the individual group in accordance with the determined degree of resemblance of the items.

8. The system of claim 1 , wherein the control circuit executing the trained machine learning model is further configured to determine the degree of resemblance of items based on at least one of a textual similarity or a visual similarity of the items.

9. The system of claim 1 , wherein the selected sample from the at least one of the plurality of groups comprises a sample of one or more images based on a rule from each of the plurality of groups.

10. The system of claim 9 , wherein the control circuit executing the trained machine learning model is further configured to group the sample of the one or more images from each of the plurality of groups into the selected sample, wherein the selected sample is output in response to a determination that an image count of the selected sample is less than a threshold.

11. The system of claim 10 , wherein the control circuit executing the trained machine learning model is further configured to:

in response to a determination that the image count of the selected sample is at least the threshold, allocate each image in the selected sample into one of the plurality of datasets based on the areas in the product storage facility images in the selected sample were captured;

for each dataset of the plurality of datasets, cluster each image in the dataset into one of the plurality of groups based on the degree of resemblance of items depicted in the image relative to those items depicted in other images in the dataset;

select a second sample from at least one of the plurality of groups; and

determine whether a second image count of the second sample is less than the threshold, wherein the allocation of each image in the selected sample into one of the plurality of datasets, the clustering of each image in the dataset into one of the plurality of groups, and the selection of a subsequent sample from at least one of the plurality of groups are repeated until the subsequent sample is less than the threshold.

12. The system of claim 9 , wherein the plurality of groups comprise a homogenous group, a heterogeneous group, a low similarity group, and an individual group, and wherein the rule comprises:

selecting, by the control circuit executing the trained machine learning model and for images included in the homogenous group, at least two or three images based on a highest number of at least one of depicted bounding boxes, boundary box aspect ratio, or Optical Character Recognition (OCR) count;

selecting, by the control circuit executing the trained machine learning model and for images included in the heterogeneous group, a predetermined percentage of the images included in the heterogeneous group; and

selecting, by the control circuit executing the trained machine learning model and for images included in the individual group, all images included in the individual group.

13. The system of claim 1 , wherein the selected sample from the dominant product identifier group comprises an image having a threshold range of depicted items being associated with a dominant product identifier, wherein the dominant product identifier is a product identifier that is most identified with the depicted items relative to other identified product identifiers of the depicted items in the image.

14. The system of claim 1 , wherein the selected sample from the non-dominant product identifier group comprises all images associated with the non-dominant product identifier group.

15. The system of claim 1 , wherein the areas comprise one of an aisle, a bin, a pallet, or a rack storing one or more office supply products, grocery products, electronic products, and household supply products.

16. A method for processing captured images of objects at a product storage facility, the method comprising:

storing, by a memory, a plurality of images, the plurality of images comprising first images and second images, wherein each of the first images contain items not detected by a trained machine learning model as being associated with a recognized product identifier, and wherein each of the second images contain items detected by the trained machine learning model as being associated with multiple recognized product identifiers;

allocating, by a control circuit executing the trained machine learning model, each of the first images into one of a plurality of datasets based on areas in the product storage facility the first images were captured;

for each dataset of the plurality of datasets, clustering, by the control circuit executing the trained machine learning model, each image in the dataset into one of a plurality of groups based on a degree of resemblance of items depicted in the image relative to those items depicted in other images in the dataset;

selecting, by the control circuit executing the trained machine learning model, a sample from at least one of the plurality of groups;

clustering, by the control circuit executing the trained machine learning model, each of the second images into one of a dominant product identifier group and a non-dominant product identifier group;

selecting, by the control circuit executing the trained machine learning model, a sample from the dominant product identifier group;

selecting, by the control circuit executing the trained machine learning model, a sample from the non-dominant product identifier group; and

outputting, by the control circuit executing the trained machine learning model, at least one of the selected sample from the at least one of the plurality of groups, the selected sample from the dominant product identifier group, and the selected sample from the non-dominant product identifier group to be used to retrain the trained machine learning model.

17. The method of claim 16 , wherein the plurality of groups comprise a homogenous group, a heterogeneous group, a low similarity group, and an individual group, wherein the homogenous group includes images where the degree of resemblance of items depicted in each image is greater than 70%, wherein the low similarity group includes images where the degree of resemblance of items depicted in each image is less than 30%, wherein the heterogeneous group includes images where the degree of resemblance of items depicted in each image is at least 30% but no more than 70%, and wherein the individual group includes images that do not have resemblance with any other image in the dataset.

18. The method of claim 16 , further comprising determining, by the control circuit executing the trained machine learning model, the degree of resemblance of items based on at least one of a textual similarity or a visual similarity of the items.

19. The method of claim 16 , wherein the selected sample from the at least one of the plurality of groups comprises a sample of one or more images based on a rule from each of the plurality of groups.

20. The method of claim 19 , further comprising grouping, by the control circuit executing the trained machine learning model, the sample of the one or more images from each of the plurality of groups into the selected sample, wherein the selected sample is output in response to a determination that an image count of the selected sample is less than a threshold.

Assignments (3)

Continuity (1)

References Cited (141)