IP Library Granted Patent US 12,469,255
Granted Patent B2
US 12,469,255 · App. 18/168,198 · Granted Nov 11, 2025

Systems and methods for identifying different product identifiers that correspond to the same product

Inventors: Ashlin Ghosh (Ernakulam, IN); Feiyun Zhu (Allen, TX); Avinash M. Jade (Bangalore, IN); Lingfeng Zhang (Dallas, TX); Amit Jhunjhunwala (Bangalore, IN); Raghava Balusu (Achanta, IN); William Craig Robinson, Jr. (Centerton, AR); Benjamin R. Ellison (San Francisco, CA); Srinivas Muktevi (Bengaluru, IN); Zhaoliang Duan (Frisco, TX)
Assignee: Walmart Apollo, LLC
G06V10/762G06Q10/087G06V10/761G06V2201/07
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,469,255
App. No.
18/168,198
Granted
Nov 11, 2025
Kind
B2
Abstract

In some embodiments, apparatuses and methods are provided herein useful to processing captured images. In some embodiments, there is provided a system for processing captured images of objects at a product storage facility including a trained machine learning model; and a control circuit. The control circuit may group a plurality of product identifiers into one or more clusters based on at least one of visual similarity of corresponding images, textual similarity of corresponding associated descriptions, and associated relationships between product identifiers of the plurality of product identifiers; determine clusters having common elements that are at least within a similarity threshold of each other; merge the clusters with the common elements; and generate a mapping dataset used to retrain the trained machine learning model to identify a plurality of objects. The mapping dataset may include a plurality of associations of associated product identifiers to a single object.

Claims (37)

1 . A system for processing captured images of objects at a product storage facility, the system comprising:

one or more image capture devices configured to capture images of the objects at the product storage facility;

a trained machine learning model stored in a memory, wherein the trained machine learning model is trained on the images and metadata associated with a plurality of product identifiers, a plurality of product reference images, a plurality of planogram data, or combination thereof; and

a control circuit executing the trained machine learning model configured to:

convert the captured images into a plurality tensors;

group the plurality of product identifiers into one or more clusters based on at least one of visual similarity of corresponding images, textual similarity of corresponding associated descriptions, and associated relationships between product identifiers of the plurality of product identifiers, wherein the visual similarity of the corresponding images is determined based on a calculation of hamming distances pairwise among the plurality of tensors;

determine clusters having common elements that are at least within a similarity threshold of each other;

merge the clusters with the common elements; and

generate a mapping dataset used to retrain the trained machine learning model to identify a plurality of objects, wherein the mapping dataset comprises a plurality of associations of associated product identifiers to a single object.

2 . The system of claim 1 , further comprising

a database configured to store the images.

3 . The system of claim 2 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

4 . The system of claim 1 , wherein the visual similarity of the corresponding images is determined based on a calculation of hamming distances between the corresponding images and a grouping of images having corresponding hamming distances that are less than a distance threshold.

5 . The system of claim 1 , wherein the textual similarity of the corresponding associated descriptions is determined based on a match of a predefined text associated with the corresponding associated descriptions.

6 . The system of claim 1 , wherein the product storage facility comprises at least one of: a product distribution center, a fulfillment center, and a retail store.

7 . The system of claim 1 , wherein the common elements comprise one or more product identifiers.

8 . The system of claim 1 , wherein the control circuit executing the trained machine learning model is further configured to generate a second mapping dataset used to retrain the trained machine learning model to identify the plurality of objects, wherein the second mapping dataset comprises a plurality of associations of each of the associated product identifiers to at least one corresponding product storage facility.

9 . The system of claim 1 , wherein the associated relationships comprise different product identifiers that are variants of same object.

10 . The system of claim 9 , wherein the variants of the same object comprise differences in at least one of: size, color, and pattern.

11 . A method for processing captured images of objects at a product storage facility, the method comprising:

capturing, by one or more image capture devices, images of the objects at the product storage facility;

training a machine learning model on the images and metadata associated with a plurality of product identifiers, a plurality of product reference images, a plurality of planogram data, or combination thereof;

converting, by a control circuit, the captures images into a plurality of tensors;

grouping, by the control circuit executing the trained machine learning model stored in a memory, the plurality of product identifiers into one or more clusters based on at least one of visual similarity of corresponding images, textual similarity of corresponding associated descriptions, and associated relationships between product identifiers of the plurality of product identifiers, wherein the visual similarity of the corresponding images is determined based on a calculation of hamming distances pairwise among the plurality of tensors;

determining, by the control circuit executing the trained machine learning model, clusters having common elements that are at least within a similarity threshold of each other;

merging, by the control circuit executing the trained machine learning model, the clusters with the common elements; and

generating, by the control circuit executing the trained machine learning model, a mapping dataset used to retrain the trained machine learning model to identify a plurality of objects, wherein the mapping dataset comprises a plurality of associations of associated product identifiers to a single object.

12 . The method of claim 11 , further comprising

storing, by a database, the images.

13 . The method of claim 12 , wherein at least one of the one or more image capture devices is coupled to a motorized robotic unit.

14 . The method of claim 12 , wherein the visual similarity of the corresponding images is determined based on a calculation of hamming distances between the corresponding images and a grouping of images having corresponding hamming distances that are less than a distance threshold.

15 . The method of claim 11 , wherein the textual similarity of the corresponding associated descriptions is determined based on a match of a predefined text associated with the corresponding associated descriptions.

16 . The method of claim 11 , wherein the product storage facility comprises at least one of: a product distribution center, a fulfillment center, and a retail store.

17 . The method of claim 11 , wherein the common elements comprise one or more product identifiers.

18 . The method of claim 11 , further comprising generating, by the control circuit executing the trained machine learning model, a second mapping dataset used to retrain the trained machine learning model to identify the plurality of objects, wherein the second mapping dataset comprises a plurality of associations of each of the associated product identifiers to at least one corresponding product storage facility.

19 . The method of claim 11 , wherein the associated relationships comprise different product identifiers that are variants of same object.

20 . The method of claim 19 , wherein the variants of the same object comprise differences in at least one of: size, color, and pattern.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 11, 2023
From: WM GLOBAL TECHNOLOGY SERVICES INDIA PRIVATE LIMITED
To: WALMART APOLLO, LLC
Reel/Frame 064568/0994 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 14, 2023
From: GHOSH, ASHLIN; JADE, AVINASH M.; JHUNJHUNWALA, AMIT; BALUSU, RAGHAVA; MUKTEVI, SRINIVAS
To: WM GLOBAL TECHNOLOGY SERVICES INDIA PRIVATE LIMITED
Reel/Frame 062695/0819 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 14, 2023
From: ZHU, FEIYUN; ZHANG, LINGFENG; ROBINSON, WILLIAM CRAIG, JR.; ELLISON, BENJAMIN R.; DUAN, ZHAOLIANG
To: WALMART APOLLO, LLC
Reel/Frame 062698/0195 →
Continuity (1)
Related Publication 20240273863A1 · Aug 15, 2024
References Cited (153)
US 5074594A · Laganowski · 1991 [cited by applicant]
US 6570492B1 · Peratoner · 2003 [cited by applicant]
US 8165407B1 · Khosla et al. · 2012 [cited by applicant]
US 8352465B1 · Jing · 2013 [cited by examiner]
US 8700494B2 · Carlson · 2014 [cited by applicant]
US 8923650B2 · Wexler · 2014 [cited by applicant]
US 8965104B1 · Hickman · 2015 [cited by applicant]
US 9275308B2 · Szegedy · 2016 [cited by applicant]
US 9477955B2 · Goncalves · 2016 [cited by applicant]
US 9526127B1 · Taubman · 2016 [cited by applicant]
US 9576310B2 · Cancro · 2017 [cited by applicant]
US 9659204B2 · Wu · 2017 [cited by applicant]
US 9811754B2 · Schwartz · 2017 [cited by applicant]
US 10002344B2 · Wu · 2018 [cited by applicant]
US 10019803B2 · Venable · 2018 [cited by applicant]
US 10032072B1 · Tran · 2018 [cited by applicant]
US 10129524B2 · Ng · 2018 [cited by applicant]
US 10210432B2 · Pisoni · 2019 [cited by applicant]
US 10373116B2 · Medina · 2019 [cited by applicant]
US 10445569B1 · Lin et al. · 2019 [cited by applicant]
US 10572757B2 · Graham · 2020 [cited by applicant]
US 10592854B2 · Schwartz · 2020 [cited by applicant]
US 10664722B1 · Sharma et al. · 2020 [cited by applicant]
US 10796352B2 · Chechuy · 2020 [cited by applicant]
US 10839452B1 · Guo · 2020 [cited by applicant]
US 10922574B1 · Tariq · 2021 [cited by applicant]
US 10943278B2 · Benkreira · 2021 [cited by applicant]
US 10956711B2 · Adato · 2021 [cited by applicant]
US 10990950B2 · Garner · 2021 [cited by applicant]
US 10991036B1 · Bergstrom · 2021 [cited by applicant]
US 11036949B2 · Powell · 2021 [cited by applicant]
US 11055905B2 · Tagra · 2021 [cited by applicant]
US 11087272B2 · Skaff · 2021 [cited by applicant]
US 11151426B2 · Dutta · 2021 [cited by applicant]
US 11163805B2 · Arocho · 2021 [cited by applicant]
US 11276034B2 · Shah · 2022 [cited by applicant]
US 11282287B2 · Gausebeck · 2022 [cited by applicant]
US 11295163B1 · Schoner · 2022 [cited by applicant]
US 11308775B1 · Sinha · 2022 [cited by applicant]
US 11409977B1 · Glaser · 2022 [cited by applicant]
US 20050238465A1 · Razumov · 2005 [cited by applicant]
US 20100188580A1 · Paschalaks · 2010 [cited by applicant]
US 20110040427A1 · Ben-Tzvi · 2011 [cited by applicant]
US 20120303412A1 · Etzioni · 2012 [cited by applicant]
US 20140002239A1 · Rayner · 2014 [cited by applicant]
US 20140247116A1 · Davidson · 2014 [cited by applicant]
US 20140307938A1 · Doi · 2014 [cited by applicant]
US 20150363660A1 · Vidal · 2015 [cited by applicant]
US 20160203525A1 · Hara · 2016 [cited by applicant]
US 20170106738A1 · Gillett · 2017 [cited by applicant]
US 20170286773A1 · Skaff · 2017 [cited by applicant]
US 20180005176A1 · Williams · 2018 [cited by applicant]
US 20180018788A1 · Olmstead · 2018 [cited by applicant]
US 20180108134A1 · Venable · 2018 [cited by applicant]
US 20180197223A1 · Grossman · 2018 [cited by applicant]
US 20180260772A1 · Chaubard · 2018 [cited by applicant]
US 20190025849A1 · Dean · 2019 [cited by applicant]
US 20190043003A1 · Fisher · 2019 [cited by applicant]
US 20190050932A1 · Dey · 2019 [cited by applicant]
US 20190087772A1 · Medina · 2019 [cited by applicant]
US 20190163698A1 · Kwon · 2019 [cited by applicant]
US 20190197561A1 · Adato · 2019 [cited by applicant]
US 20190220482A1 · Crosby · 2019 [cited by applicant]
US 20190236531A1 · Adato · 2019 [cited by applicant]
US 20200118063A1 · Fu · 2020 [cited by applicant]
US 20200193206A1 · Turkelson et al. · 2020 [cited by applicant]
US 20200242392A1 · Scott et al. · 2020 [cited by applicant]
US 20200246977A1 · Swietojanski · 2020 [cited by applicant]
US 20200265494A1 · Glaser · 2020 [cited by applicant]
US 20200324976A1 · Diehr · 2020 [cited by applicant]
US 20200356813A1 · Sharma · 2020 [cited by applicant]
US 20200380226A1 · Rodriguez · 2020 [cited by applicant]
US 20200387858A1 · Hasan · 2020 [cited by applicant]
US 20210049541A1 · Gong · 2021 [cited by applicant]
US 20210049542A1 · Dalal · 2021 [cited by applicant]
US 20210125001A1 · Guo · 2021 [cited by examiner]
US 20210142097A1 · Zheng et al. · 2021 [cited by applicant]
US 20210142105A1 · Siskind · 2021 [cited by applicant]
US 20210150231A1 · Kehl · 2021 [cited by applicant]
US 20210192780A1 · Kulkarni · 2021 [cited by applicant]
US 20210216954A1 · Chaubard · 2021 [cited by applicant]
US 20210272269A1 · Suzuki · 2021 [cited by applicant]
US 20210319684A1 · Ma · 2021 [cited by applicant]
US 20210342914A1 · Dalal · 2021 [cited by applicant]
US 20210400195A1 · Adato · 2021 [cited by applicant]
US 20210406812A1 · Deshmukh · 2021 [cited by applicant]
US 20220043547A1 · Jahjah · 2022 [cited by applicant]
US 20220051179A1 · Savvides · 2022 [cited by applicant]
US 20220058425A1 · Savvides · 2022 [cited by applicant]
US 20220067085A1 · Nihas · 2022 [cited by applicant]
US 20220114403A1 · Shaw · 2022 [cited by applicant]
US 20220114537A1 · Griffin · 2022 [cited by examiner]
US 20220114821A1 · Arroyo · 2022 [cited by applicant]
US 20220121852A1 · Curtis · 2022 [cited by examiner]
US 20220138914A1 · Wang · 2022 [cited by applicant]
US 20220165074A1 · Srivastava · 2022 [cited by applicant]
US 20220222924A1 · Pan · 2022 [cited by applicant]
US 20220262008A1 · Kidd · 2022 [cited by applicant]
US 20230078604A1 · Rein · 2023 [cited by examiner]
US 20240119409A1 · Balusu · 2024 [cited by examiner]
US 20240249506A1 · Arora · 2024 [cited by examiner]
US 20240257047A1 · Balusu · 2024 [cited by examiner]
US 20240265565A1 · Zhang · 2024 [cited by examiner]
US 20240273463A1 · Pachauri · 2024 [cited by examiner]
CN 106347550B · 2019 [cited by applicant]
CN 110348439B · 2019 [cited by applicant]
CN 110443298B · 2022 [cited by applicant]
CN 114898358A · 2022 [cited by applicant]
CN 115205584A · 2022 [cited by applicant]
EP 3217324A1 · 2017 [cited by applicant]
EP 3437031 · 2019 [cited by applicant]
EP 3479298 · 2019 [cited by applicant]
WO 2006113281A2 · 2006 [cited by applicant]
WO 2017201490A1 · 2017 [cited by applicant]
WO 2018093796 · 2018 [cited by applicant]
WO 2020051213A1 · 2020 [cited by applicant]
WO WO2021152070A1 · 2021 [cited by examiner]
WO 2021186176A1 · 2021 [cited by applicant]
WO 2021247420A2 · 2021 [cited by applicant]
Klasson et al, Using Variational Multi-view Learning for Classification of Grocery Items, Patterns 1, 100143 pp. 1-19. (Year: 2020). [cited by examiner]
Rodriquez, K., PCT International Search Report and Written Opinion, mailed May 22, 2024, in connection with International Application No. PCT/US2024/14689, all pages. [cited by applicant]
U.S. Appl. No. 17/963,751, filed Oct. 11, 2022, Yilun Chen. [cited by applicant]
U.S. Appl. No. 17/963,787, filed Oct. 11, 2022, Lingfeng Zhang. [cited by applicant]
U.S. Appl. No. 17/963,802, filed Oct. 11, 2022, Lingfeng Zhang. [cited by applicant]
U.S. Appl. No. 17/963,903, filed Oct. 11, 2022, Raghava Balusu. [cited by applicant]
U.S. Appl. No. 17/966,580, filed Oct. 14, 2022, Paarvendhan Puviyarasu. [cited by applicant]
U.S. Appl. No. 17/971,350, filed Oct. 21, 2022, Jing Wang. [cited by applicant]
U.S. Appl. No. 17/983,773, filed Nov. 9, 2022, Lingfeng Zhang. [cited by applicant]
U.S. Appl. No. 18/102,999, filed Jan. 30, 2023, Han Zhang. [cited by applicant]
U.S. Appl. No. 18/103,338, filed Jan. 30, 2023, Wei Wang. [cited by applicant]
U.S. Appl. No. 18/106,269, filed Feb. 6, 2023, Zhaoliang Duan. [cited by applicant]
U.S. Appl. No. 18/158,925, filed Jan. 24, 2023, Raghava Balusu. [cited by applicant]
U.S. Appl. No. 18/158,950, filed Jan. 24, 2023, Ishan Arora. [cited by applicant]
U.S. Appl. No. 18/158,969, filed Jan. 24, 2023, Zhaoliang Duan. [cited by applicant]
U.S. Appl. No. 18/158,983, filed Jan. 24, 2023, Ashlin Ghosh. [cited by applicant]
U.S. Appl. No. 18/161,788, filed Jan. 30, 2023, Raghava Balusu. [cited by applicant]
U.S. Appl. No. 18/165,152, filed Feb. 2, 2023, Han Zhang. [cited by applicant]
U.S. Appl. No. 18/168,174, filed Feb. 13, 2023, Abhinav Pachauri. [cited by applicant]
Chaudhuri, Abon et al.; “A Smart System for Selection of Optimal Product Images in E-Commerce”; 2018 IEEE Conference on Big Data (Big Data); Dec. 10-13, 2018; IEEE; <https://ieeexplore.ieee.org/document/8622259>; pp. 17… [cited by applicant]
Chenze, Brandon et al.; “Iterative Approach for Novel Entity Recognition of Foods in Social Media Messages”; 2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI); Aug. 9-11… [cited by applicant]
Kaur, Ramanpreet et al.; “A Brief Review on Image Stitching and Panorama Creation Methods”; International Journal of Control Theory and Applications; 2017; vol. 10, No. 28; International Science Press; Gurgaon, India; <… [cited by applicant]
Naver Engineering Team; “Auto-classification of Naver Shopping Product Categories using TensorFlow”; <https://blog.tensorflow.org/2019/05/auto-classification-of-naver-shopping.html>; May 20, 2019; pp. 1-15. [cited by applicant]
Paolanti, Marine et al.; “Mobile robot for retail surveying and inventory using visual and textual analysis of monocular pictures based on deep learning”; European Conference on Mobile Robots; Sep. 2017, 6 pages. [cited by applicant]
Refills; “Final 3D object perception and localization”; European Commision, Dec. 31, 2016, 16 pages. [cited by applicant]
Retech Labs; “Storx | RetechLabs”; <https://retechlabs.com/storx/>; available at least as early as Jun. 22, 2019; retrieved from Internet Archive Wayback Machine <https://web.archive.org/web/20190622012152/https://retec… [cited by applicant]
Schroff, Florian et al.; “Facenet: a unified embedding for face recognition and clustering”; 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Jun. 7-12, 2015; IEEE; <https://ieeexplore.ieee.org/do… [cited by applicant]
Tan, Mingxing et al.; “EfficientDet: Scalable and Efficient Object Detection”; 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Jun. 13-19, 2020; IEEE; <https://ieeexplore.ieee.org/document/91… [cited by applicant]
Tan, Mingxing et al.; “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”; Proceedings of the 36th International Conference on Machine Learning; 2019; vol. 97; PLMR; <http://proceedings.mlr.press/… [cited by applicant]
Technology Robotix Society; “Colour Detection”; <https://medium.com/image-processing-in-robotics/colour-detection-e15bc03b3f61>; Jul. 2, 2019; pp. 1-6. [cited by applicant]
Tonioni, Alessio et al.; “A deep learning pipeline for product recognition on store shelves”; 2018 IEEE International Conference on Image Processing, Applications and Systems (IPAS); Dec. 12-14, 2018; IEEE; <https://iee… [cited by applicant]
Trax Retail; “Image Recognition Technology for Retail | Trax”; <https://traxretail.com/retail/>; available at least as early as Apr. 20, 2021; retrieved from Internet Wayback Machine <https://web.archive.org/web/2021042… [cited by applicant]
Verma, Nishchal, et al.; “Object identification for inventory management using convolutional neural network”; IEEE Applied Imagery Pattern Recognition Workshop (AIPR); Oct. 2016, 6 pages. [cited by applicant]
Zhang, Jicun, et al.; “An Improved Louvain Algorithm for Community Detection”; Advanced Pattern and Structure Discovery from Complex Multimedia Data Environments 2021; Nov. 23, 2021; Mathematical Problems in Engineering… [cited by applicant]