IP Library Granted Patent US 12,367,457
Granted Patent B2
US 12,367,457 · App. 17/983,773 · Granted Jul 22, 2025

Systems and methods of verifying price tag label-product pairings

Inventors: Lingfeng Zhang (Dallas, TX); Wei Wang (Dallas, TX); Avinash Madhusudanrao Jade (Bangalore, IN); Raghava Balusu (Achanta, IN); Mingquan Yuan (Flower Mound, TX); Jing Wang (Dallas, TX); Han Zhang (Allen, TX); Zhaoliang Duan (Frisco, TX); William Craig Robinson, Jr. (Centerton, AR); Benjamin R. Ellison (San Francisco, CA); Paul Lewis Lobo (Irving, TX)
Assignee: Walmart Apollo, LLC
G06Q10/087G06V20/50G06V30/19007G06V30/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,367,457
App. No.
17/983,773
Granted
Jul 22, 2025
Kind
B2
Abstract

Systems and methods of verifying pairings of price tag labels and products at a product storage facility include an image capture device that moves about and captures images of product storage structures at the product storage facility, and a database that stores inventory data associated with the products stocked at the product storage facility. A computing device processes the images to extract one or more characters from the price tag labels and products detected in the images, correlates the extracted characters to the inventory data to identify a product code and a product name that matches the characters extracted from the price tag labels and the products. If the characters extracted from the price tag and from the on-shelf product to which the price tag label is allocated match, the database is updated to reflect that the price tag label is properly allocated to the appropriate on-shelf product.

Claims (45)

1. A system for verifying pairings of on-shelf tag labels and products at a product storage facility, the system comprising:

an electronic database configured to store inventory data for the product storage facility;

a device configured for movement throughout the product storage facility;

a single camera incorporated into the device, the single camera configured to capture a single image of a product storage structure in the product storage facility, wherein the single camera has a field of view that is wide enough to capture the entirety of the product storage structure in the single image, and wherein the product storage structure is a free-standing product display that is independent from other product storage structures in the product storage facility; and

a computing device communicatively coupled to the single camera, the computing device configured to:

detect, in the single image, a price tag label affixed to the product storage structure and a product arranged on the product storage structure;

extract character(s) from the product in the single image and character(s) from the price tag label in the single image, wherein the character(s) extracted from the product are independent from a bar code on the product and the character(s) extracted from the price tag label are independent from a bar code on the price tag label;

determine, based on the character(s) extracted from the price tag label, a product code associated with the price tag label, the product code being identified without referencing the bar code on the price tag label;

determine, based on the character(s) extracted from the product, a keyword instance indicating both a keyword on the product and a specific location of the keyword on the product;

correlate the keyword instance with the inventory data stored in the electronic database to identify a product name associated with the product, the product name being identified without referencing the bar code on the product;

determine that the product name associated with the product matches the product code associated with the price tag label; and

transmit, to the electronic database, a signal indicating that the price tag label is properly allocated to the product storage structure responsive to determining that the product name associated with the product matches the product code associated with the price tag label.

2. The system of claim 1 , wherein the computing device is further configured to extract graphical object(s) from the product detected in the single image, the product name being identified by correlating both the graphical object(s) extracted from the product and the keyword instance with the inventory data stored in the electronic database.

3. The system of claim 2 , wherein the graphical object(s) extracted from the product are independent from the bar code on the product.

4. The system of claim 2 , wherein the graphical object(s) extracted from the product are independent from the character(s) extracted from the product.

5. The system of claim 2 , wherein the computing device determines the keyword instance by generating a virtual boundary around the character(s) extracted from the product and performing optical character recognition within the virtual boundary to identify the keyword.

6. The system of claim 5 , wherein the virtual boundary excludes the graphical object(s) extracted from the product and the bar code on the product.

7. A method for verifying pairings of on-shelf labels and products at a product storage facility, the method comprising:

capturing, via a single camera incorporated into a device configured for movement throughout the product storage facility, a single image of a product storage structure in the product storage facility, wherein the single camera has a field of view that is wide enough to capture the entirety of the product storage structure in the single image, and wherein the product storage structure is a free-standing product display that is independent from other product storage structures in the product storage facility;

detecting, in the single image, a price tag label affixed to the product storage structure and a product arranged on the product storage structure;

extracting character(s) from the product in the single image and character(s) from the price tag label in the single image, wherein the character(s) extracted from the product are independent from a bar code on the product and the character(s) extracted from the price tag label are independent from a bar code on the price tag label;

determining, based on the character(s) extracted from the price tag label, a product code associated with the price tag label, the product code being identified without referencing the bar code on the price tag label;

determining, based on the character(s) extracted from the product, a keyword instance indicating both a keyword on the product and a specific location of the keyword on the product;

correlating the keyword instance with inventory data stored in an electronic database to identify a product name associated with the product, the product name being identified without referencing the bar code on the product;

determining that the product name associated with the product matches the product code associated with the price tag label; and

transmitting, to the electronic database, a signal indicating that the price tag label is properly allocated to the product storage structure responsive to determining that product name associated with the product matches the product code associated with the price tag label.

8. The method of claim 7 , further comprising extracting graphical object(s) from the product detected in the single image, the product name being identified by correlating both the graphical object(s) extracted from the product and the keyword instance with the inventory data stored in the electronic database.

9. The method of claim 8 , wherein the graphical object(s) extracted from the product are independent from the bar code on the product.

10. The method of claim 8 , wherein the graphical object(s) extracted from the product are independent from the character(s) extracted from the product.

11. The method of claim 8 , wherein determining the keyword instance comprises generating a virtual boundary around the character(s) extracted from the product and performing optical character recognition within the virtual boundary to identify the keyword.

12. The method of claim 11 , wherein the virtual boundary excludes the graphical object(s) extracted from the product and the bar code on the product.

13. A method for verifying pairings of on-shelf labels and products at a product storage facility, the method comprising:

receiving a single image of a product storage structure in the product storage facility, the single image having been captured by a single camera having a field of view wide enough to capture the entirety of the product storage structure in the single image, wherein the single camera is incorporated into a device moving throughout the product storage facility, and wherein the product storage structure is a free-standing product display that is independent from other product storage structures in the product storage facility;

detecting, in the single image, a price tag label affixed to the product storage structure and a product arranged on the product storage structure;

extracting character(s) from the product in the single image and character(s) from the price tag label in the single image, wherein the character(s) extracted from the product are independent from a bar code on the product and the character(s) extracted from the price tag label are independent from a bar code on the price tag label;

determining, based on the character(s) extracted from the price tag label, a product code associated with the price tag label, the product code being identified without referencing the bar code on the price tag label;

determining, based on the character(s) extracted from the product, a keyword instance indicating both a keyword on the product and a specific location of the keyword on the product;

correlating the keyword instance with inventory data stored in an electronic database to identify a product name associated with the product, the product name being identified without referencing the bar code on the product;

determining that the product name associated with the product matches the product code associated with the price tag label; and

transmitting, to the electronic database, a signal indicating that the price tag label is properly allocated to the product storage structure responsive to determining that the product name associated with the product matches the product code associated with the price tag label.

14. The method of claim 13 , further comprising extracting graphical object(s) from the product detected in the single image, the product name being identified by correlating both the graphical object(s) extracted from the product and the keyword instance with the inventory data stored in the electronic database.

15. The method of claim 14 , wherein the graphical object(s) extracted from the product are independent from the bar code on the product.

16. The method of claim 14 , wherein the graphical object(s) extracted from the product are independent from the character(s) extracted from the product.

17. The method of claim 14 , wherein determining the keyword instance comprises generating a virtual boundary around the character(s) extracted from the product and performing optical character recognition within the virtual boundary to identify the keyword.

18. The method of claim 17 , wherein the virtual boundary excludes the graphical object(s) extracted from the product and the bar code on the product.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 11, 2023
From: WM GLOBAL TECHNOLOGY SERVICES INDIA PRIVATE LIMITED
To: WALMART APOLLO, LLC
Reel/Frame 064567/0665 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 17, 2023
From: ZHANG, LINGFENG; WANG, WEI; YUAN, MINGQUAN; WANG, JING; ZHANG, HAN; DUAN, ZHAOLIANG; ROBINSON, WILLIAM CRAIG, JR; ELLISON, BENJAMIN R.; LOBO, PAUL LEWIS
To: WALMART APOLLO, LLC
Reel/Frame 063661/0328 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2023
From: JADE, AVINASH MADHUSUDANRAO; BALUSU, RAGHAVA
To: WM GLOBAL TECHNOLOGY SERVICES INDIA PRIVATE LIMITED
Reel/Frame 063661/0310 →
Continuity (1)
Related Publication 20240152863A1 · May 9, 2024
References Cited (143)
US 5074594A · Laganowski · 1991 [cited by applicant]
US 6570492B1 · Peratoner · 2003 [cited by applicant]
US 8630924B2 · Groenevelt · 2014 [cited by examiner]
US 8923650B2 · Wexler · 2014 [cited by applicant]
US 8965104B1 · Hickman · 2015 [cited by applicant]
US 9275308B2 · Szegedy · 2016 [cited by applicant]
US 9477955B2 · Goncalves · 2016 [cited by applicant]
US 9526127B1 · Taubman · 2016 [cited by applicant]
US 9576310B2 · Cancro · 2017 [cited by applicant]
US 9659204B2 · Wu · 2017 [cited by applicant]
US 9811754B2 · Schwartz · 2017 [cited by applicant]
US 10002344B2 · Wu · 2018 [cited by applicant]
US 10019803B2 · Venable · 2018 [cited by applicant]
US 10032072B1 · Tran · 2018 [cited by applicant]
US 10129524B2 · Ng · 2018 [cited by applicant]
US 10210432B2 · Pisoni · 2019 [cited by applicant]
US 10373116B2 · Medina · 2019 [cited by applicant]
US 10572757B2 · Graham · 2020 [cited by applicant]
US 10592854B2 · Schwartz · 2020 [cited by applicant]
US 10839452B1 · Guo · 2020 [cited by applicant]
US 10922574B1 · Tariq · 2021 [cited by applicant]
US 10943278B2 · Benkreira · 2021 [cited by applicant]
US 10956711B2 · Adato · 2021 [cited by applicant]
US 10990950B2 · Garner · 2021 [cited by applicant]
US 10991036B1 · Bergstrom · 2021 [cited by applicant]
US 11036949B2 · Powell · 2021 [cited by applicant]
US 11055905B2 · Tagra · 2021 [cited by applicant]
US 11087272B2 · Skaff · 2021 [cited by applicant]
US 11151426B2 · Dutta · 2021 [cited by applicant]
US 11163805B2 · Arocho · 2021 [cited by applicant]
US 11276034B2 · Shah · 2022 [cited by applicant]
US 11282287B2 · Gausebeck · 2022 [cited by applicant]
US 11295163B1 · Schoner · 2022 [cited by examiner]
US 11308775B1 · Sinha · 2022 [cited by applicant]
US 11409977B1 · Glaser · 2022 [cited by applicant]
US 11593755B2 · Bogolea · 2023 [cited by examiner]
US 11900653B1 · Schoner · 2024 [cited by examiner]
US 20050238465A1 · Razumov · 2005 [cited by applicant]
US 20110040427A1 · Ben-Tzvi · 2011 [cited by applicant]
US 20140002239A1 · Rayner · 2014 [cited by applicant]
US 20140247116A1 · Davidson · 2014 [cited by applicant]
US 20140307938A1 · Doi · 2014 [cited by applicant]
US 20150363660A1 · Vidal · 2015 [cited by applicant]
US 20160203525A1 · Hara · 2016 [cited by applicant]
US 20170106738A1 · Gillett · 2017 [cited by applicant]
US 20170286773A1 · Skaff · 2017 [cited by applicant]
US 20180005176A1 · Williams · 2018 [cited by applicant]
US 20180018788A1 · Olmstead · 2018 [cited by applicant]
US 20180130119A1 · Korac · 2018 [cited by examiner]
US 20180197223A1 · Grossman · 2018 [cited by applicant]
US 20180260772A1 · Chaubard · 2018 [cited by applicant]
US 20190025849A1 · Dean · 2019 [cited by applicant]
US 20190034864A1 · Skaff · 2019 [cited by examiner]
US 20190043003A1 · Fisher · 2019 [cited by applicant]
US 20190050932A1 · Dey · 2019 [cited by applicant]
US 20190087772A1 · Medina · 2019 [cited by applicant]
US 20190163698A1 · Kwon · 2019 [cited by applicant]
US 20190197561A1 · Adato · 2019 [cited by examiner]
US 20190220482A1 · Crosby · 2019 [cited by applicant]
US 20190236531A1 · Adato · 2019 [cited by applicant]
US 20190354923A1 · Taira · 2019 [cited by examiner]
US 20200005225A1 · Chaubard · 2020 [cited by examiner]
US 20200184170A1 · Reza · 2020 [cited by examiner]
US 20200246977A1 · Swietojanski · 2020 [cited by applicant]
US 20200265494A1 · Glaser · 2020 [cited by applicant]
US 20200311659A1 · Yonezawa · 2020 [cited by examiner]
US 20200324976A1 · Diehr · 2020 [cited by applicant]
US 20200356813A1 · Sharma · 2020 [cited by applicant]
US 20200380226A1 · Rodriguez · 2020 [cited by applicant]
US 20200387858A1 · Hasan · 2020 [cited by applicant]
US 20210049541A1 · Gong · 2021 [cited by applicant]
US 20210049542A1 · Dalal · 2021 [cited by applicant]
US 20210142105A1 · Siskind · 2021 [cited by applicant]
US 20210150231A1 · Kehl · 2021 [cited by applicant]
US 20210158278A1 · Bogolea · 2021 [cited by examiner]
US 20210192780A1 · Kulkarni · 2021 [cited by applicant]
US 20210216954A1 · Chaubard · 2021 [cited by applicant]
US 20210248547A1 · Bizoara · 2021 [cited by examiner]
US 20210272269A1 · Suzuki · 2021 [cited by applicant]
US 20210319684A1 · Ma · 2021 [cited by applicant]
US 20210342914A1 · Dalal · 2021 [cited by applicant]
US 20210365877A1 · Huie · 2021 [cited by examiner]
US 20210400195A1 · Adato · 2021 [cited by applicant]
US 20220043547A1 · Jahjah · 2022 [cited by applicant]
US 20220051179A1 · Savvides · 2022 [cited by applicant]
US 20220058425A1 · Savvides · 2022 [cited by applicant]
US 20220067085A1 · Nihas · 2022 [cited by applicant]
US 20220114403A1 · Shaw · 2022 [cited by applicant]
US 20220114821A1 · Arroyo · 2022 [cited by applicant]
US 20220138674A1 · Skaff · 2022 [cited by examiner]
US 20220138914A1 · Wang · 2022 [cited by applicant]
US 20220165074A1 · Srivastava · 2022 [cited by applicant]
US 20220222924A1 · Pan · 2022 [cited by applicant]
US 20220262008A1 · Kidd · 2022 [cited by applicant]
US 20220351233A1 · Momma · 2022 [cited by examiner]
US 20220391825A1 · Bogolea · 2022 [cited by examiner]
US 20230206274A1 · Lindered · 2023 [cited by examiner]
US 20230376893A1 · Bogolea · 2023 [cited by examiner]
CN 106347550B · 2019 [cited by applicant]
CN 110348439B · 2019 [cited by applicant]
CN 110443298B · 2022 [cited by applicant]
CN 114898358A · 2022 [cited by applicant]
EP 3217324A1 · 2017 [cited by applicant]
EP 3437031 · 2019 [cited by applicant]
EP 3479298 · 2019 [cited by applicant]
WO 2006113281A2 · 2006 [cited by applicant]
WO 2017201490A1 · 2017 [cited by applicant]
WO 2018093796 · 2018 [cited by applicant]
WO 2020051213A1 · 2020 [cited by applicant]
WO 2021186176A1 · 2021 [cited by applicant]
WO 2021247420A2 · 2021 [cited by applicant]
Verma, Nishchal K., et al. “Object identification for inventory management using convolutional neural network.” 2016 IEEE Applied Imagery Pattern Recognition Workshop (AIPR). IEEE, 2016. (Year: 2016). [cited by examiner]
Higa, Kyota, and Kota Iwamoto. “Robust shelf monitoring using supervised learning for improving on-shelf availability in retail stores.” Sensors 19.12 (2019): 2722. (Year: 2019). [cited by examiner]
Paolanti, Marina, et al. “Mobile robot for retail surveying and inventory using visual and textual analysis of monocular pictures based on deep learning.” 2017 European conference on mobile robots (ECMR). IEEE, 2017. (Y… [cited by examiner]
U.S. Appl. No. 17/963,751, filed Oct. 11, 2022, Yilun Chen. [cited by applicant]
U.S. Appl. No. 17/963,787, filed Oct. 11, 2022, Lingfeng Zhang. [cited by applicant]
U.S. Appl. No. 17/963,802, filed Oct. 11, 2022, Lingfeng Zhang. [cited by applicant]
U.S. Appl. No. 17/963,903, filed Oct. 11, 2022, Raghava Balusu. [cited by applicant]
U.S. Appl. No. 17/966,580, filed Oct. 14, 2022, Paarvendhan Puviyarasu. [cited by applicant]
U.S. Appl. No. 17/971,350, filed Oct. 21, 2022, Jing Wang. [cited by applicant]
U.S. Appl. No. 18/102,999, filed Jan. 30, 2023, Han Zhang. [cited by applicant]
U.S. Appl. No. 18/103,338, filed Jan. 30, 2023, Wei Wang. [cited by applicant]
U.S. Appl. No. 18/158,925, filed Jan. 24, 2023, Raghava Balusu. [cited by applicant]
U.S. Appl. No. 18/158,950, filed Jan. 24, 2023, Ishan Arora. [cited by applicant]
U.S. Appl. No. 18/158,969, filed Jan. 24, 2023, Zhaoliang Duan. [cited by applicant]
U.S. Appl. No. 18/158,983, filed Jan. 24, 2023, Ashlin Ghosh. [cited by applicant]
U.S. Appl. No. 18/161,788, filed Jan. 30, 2023, Raghava Balusu. [cited by applicant]
Chaudhuri, Abon et al.; “A Smart System for Selection of Optimal Product Images in E-Commerce”; 2018 IEEE Conference on Big Data (Big Data); Dec. 10-13, 2018; IEEE; < https://ieeexplore.ieee.org/document/8622259>; pp. 1… [cited by applicant]
Chenze, Brandon et al.; “Iterative Approach for Novel Entity Recognition of Foods in Social Media Messages”; 2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI); Aug. 9-11… [cited by applicant]
Naver Engineering Team; “Auto-classification of Naver Shopping Product Categories using TensorFlow”; <https://blog.tensorflow.org/2019/05/auto-classification-of-naver-shopping.html>; May 20, 2019; pp. 1-13. [cited by applicant]
Paolanti, Marine et al.; “Mobile robot for retail surveying and inventory using visual and textual analysis of monocular pictures based on deep learning”; European Conference on Mobile Robots; Sep. 2017, 6 pages. [cited by applicant]
Ramanpreet Kaur et al.; “A Brief Review on Image Stitching and Panorama Creation Methods”; International Journal of Control Theory and Applications; 2017; vol. 10, No. 28; International Science Press; Gurgaon, India; <h… [cited by applicant]
Refills; “Final 3D object perception and localization”; European Commision, Dec. 31, 2016, 16 pages. [cited by applicant]
Retech Labs; “Storx | RetechLabs”; <https://retechlabs.com/storx/>; available at least as early as Jun. 22, 2019; retrieved from Internet Archive Wayback Machine <https://web.archive.org/web/20190622012152/https://retec… [cited by applicant]
Schroff, Florian et al.; “Facenet: a unified embedding for face recognition and clustering”; 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Jun. 7-12, 2015; IEEE; <https://ieeexplore.ieee.org/do… [cited by applicant]
Singh, Ankit; “Automated Retail Shelf Monitoring Using AI”; <https://blog.paralleldots.com/shelf-monitoring/automated-retail-shelf-monitoring-using-ai/>; Sep. 20, 2019; pp. 1-12. [cited by applicant]
Singh, Ankit; “Image Recognition and Object Detection in Retail”; <https://blog.paralleldots.com/featured/image-recognition-and-object-detection-in-retail/>; Sep. 26, 2019; pp. 1-11. [cited by applicant]
Tan, Mingxing et al.; “EfficientDet: Scalable and Efficient Object Detection”; 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Jun. 13-19, 2020; IEEE; <https://ieeexplore.ieee.org/document/91… [cited by applicant]
Tan, Mingxing et al.; “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”; Proceedings of the 36th International Conference on Machine Learning; 2019; vol. 97; PLMR; <http://proceedings.mlr.press/… [cited by applicant]
Technology Robotix Society; “Colour Detection”; <https://medium.com/image-processing-in-robotics/colour-detection-e15bc03b3f61>; Jul. 2, 2019; pp. 1-8. [cited by applicant]
Tonioni, Alessio et al.; “A deep learning pipeline for product recognition on store shelves”; 2018 IEEE International Conference on Image Processing, Applications and Systems (IPAS); Dec. 12-14, 2018; IEEE; <https://iee… [cited by applicant]
Trax Retail; “Image Recognition Technology for Retail | Trax”; <https://traxretail.com/retail/>; available at least as early as Apr. 20, 2021; retrieved from Internet Wayback Machine <https://web.archive.org/web/2021042… [cited by applicant]
Verma, Nishchal et al.; “Object identification for inventory management using convolutional neural network”; IEEE Applied Imagery Pattern Recognition Workshop (AIPR); Oct. 2016, 6 pages. [cited by applicant]