IP Library › Granted Patent US 11,720,623
Granted Patent B2
US 11,720,623 · App. 17/097,601 · Granted Aug 8, 2023

Systems and methods for automatically annotating images

Inventors: Haining Liu (Dallas, TX); Feiyun Zhu (Arlington, TX); Ryan B. Reagan (Fort Worth, TX); Zhichun Xiao (Plano, TX)
Assignee: Walmart Apollo, LLC
G06F16/55G06F18/2148G06N20/00G06T3/0056G06T7/136G06V10/26G06V10/764G06V10/774G06V10/247
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,720,623
App. No.
17/097,601
Granted
Aug 8, 2023
Kind
B2
Abstract

In some embodiments, apparatuses and methods are provided herein useful to automatically annotating images. In some embodiments, a system for automatically annotating images comprises a database, wherein the database is configured to store images and annotations for the images and a control circuit, wherein the control circuit is communicatively coupled to the database, and wherein the control circuit is configured to retrieve, from the database, an image, generate, based on the image, a collection of augmented images, generate segmentation maps for each image in the collection of augmented images, wherein each of the segmentation maps include segments, select, based on a threshold, ones of the segments above a threshold, merge the ones of the segments above the threshold to create a segmented image, and generate, for each segment of the segmented image, classifications, wherein an annotation for the image includes the segmented images and the classifications.

Claims (43)

1. A system for automatically annotating images, the system comprising:

a database, wherein the database is configured to store images and annotations for the images; and

a control circuit, wherein the control circuit is communicatively coupled to the database, and wherein the control circuit is configured to:

retrieve, from that database, an image;

generate, based on the image, a collection of augmented images;

generate segmentation maps for each image in the collection of augmented images, wherein each of the segmentation maps include segments;

select, based on a threshold, ones of the segments above a threshold;

merge the ones of the segments above the threshold to create a segmented image; and

generate, for each segment of the segmented image, classifications, wherein an annotation for the image includes the segmented images and the classifications.

2. The system of claim 1 , further comprising:

an image capture device, wherein the image capture device is configured to capture images of carts.

3. The system of claim 2 , wherein the control circuit is further configured to:

identify, based on a machine learning model, the images, and the annotations for the images, products located in the carts.

4. The system of claim 3 , wherein the control circuit is further configured to:

compile, based on transaction histories associated with the carts, a list of products purchased with each cart, wherein the identification of the products located in the carts is further based on the list of products purchased with each cart.

5. The system of claim 2 , wherein the control circuit is further configured to:

segment the images of carts.

6. The system of claim 1 , wherein the control circuit generates the augmented images by one or more of adjusting color of the image, adjusting noise of the image, adjusting sharpness of the image, rotating the image, and cropping the image.

7. The system of claim 1 , wherein the control circuit generates the augmented images randomly.

8. The system of claim 1 , wherein the control circuit generates at least twenty augmented images based on the image.

9. The system of claim 1 , wherein the threshold is a consistency threshold representing a consistency of the segments between the segmentation maps.

10. The system of claim 9 , wherein the consistency threshold is at least 70%.

11. A method for automatically annotating images, the method comprising:

storing, in a database, images and annotations for the images;

retrieving, by a control circuit from the database, an image;

generating, by the control circuit based on the image, a collection of augmented images;

generating, by the control circuit, segmentation maps for each of the augmented images, wherein each of the segmentation maps includes segments;

selecting, by the control circuit based on a threshold, ones of the segments above the threshold;

merging, by the control circuit, the ones of the segments above the threshold to create a segmented image; and

generating, for each segment of the segmented image, classifications, wherein an annotation for the image includes the segmented image and the classifications.

12. The method of claim 11 , further comprising:

capturing, by an image capture device, images of carts.

13. The method of claim 12 , further comprising:

identifying, by the control circuit based on a machine learning model, the images, and the annotations for the images, products located in the carts.

14. The method of claim 13 , further comprising:

compiling, by the control circuit based on transaction histories associated with the carts, a list of products purchased with each cart, wherein the identifying the products located in the carts is further based on the list of products purchased with each cart.

15. The method of claim 12 , further comprising:

segmenting, by the control circuit, the images of the carts.

16. The method of claim 11 , wherein the control circuit generates the augmented images by one or more of adjusting color of the image, adjusting noise of the image, adjusting sharpness of the image, rotating the image, and cropping the image.

17. The method of claim 11 , wherein the control circuit generates the augments images randomly.

18. The method of claim 11 , wherein the control circuit generates at least twenty augmented images.

19. The method of claim 11 , wherein the threshold is a consistency threshold representing a consistency of the segments between the segmentation maps.

20. The method of claim 19 , wherein the consistency threshold is at least 70%.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 13, 2020
From: LIU, HAINING; ZHU, FEIYUN; REAGAN, RYAN B.; XIAO, ZHICHUN
To: WALMART APOLLO, LLC
Reel/Frame 054365/0672 →
Continuity (2)
Provisional Application 62935378 · Nov 14, 2019
Related Publication 20210150276A1 · May 20, 2021
Cited By (1)
US 12,657,686