Systems and methods of detecting price tags and associating the price tags with products
Systems and methods of analyzing on-shelf price tag labels and products at a product storage facility include an image capture device that captures one or more images of one or more product storage structures at a product storage facility. A computing device communicatively coupled to the image capture device analyzes the images of the product storage structures captured by the image capture device and detects individual price tag labels and individual products located on the product storage structure. Based on the detection of the price tag labels and the products, the computing device also defines separate product storage spaces of the product storage structure, determines which price tag labels are allocated to which of the separate product storage spaces, and associates in a database the price tag labels allocated to the product storage spaces with the products stored in those product storage spaces.
1. A system for analyzing on-shelf price tag labels and products at a product storage facility, the system comprising:
a motorized robotic unit, wherein the motorized robotic unit includes wheels permitting the motorized robotic unit to move about the product storage facility;
an image capture device, implemented on the motorized robotic unit, having a field of view that includes at least a portion of a product storage structure at the product storage facility having the products arranged thereon, wherein the image capture device is configured to capture one or more images from one or more angles of at least the portion of the product storage structure;
one or more computer vision models communicatively coupled to a control circuit of a computing device, wherein the one or more computer vision models are trained using data sets including at least raw image data collected from the captured one or more images, reference image data associated with reference images of one or more product storage structures, reference images of one or more products, reference images of one or more price tag labels, and planogram data associated with the product storage facility; and
the computing device including the control circuit, the computing device being communicatively coupled to the image capture device, the control circuit being configured to use the one or more computer vision models to:
analyze the at least one image of the one or more images of at least the portion of the product storage structure captured by the image capture device to detect individual ones of one or more price tag labels located on the product storage structure within the at least one image;
analyze the at least one image of at least the portion of the product storage structure to detect individual ones of the one or more products located on the product storage structure;
based on the detection of the individual ones of the one more on-shelf products located on at least the portion of the product storage structure, define one or more separate product storage spaces of the product storage structure, wherein each of the one or more separate product storage spaces contains a group of one or more identical products therein;
based on detection of the individual ones of the one or more on-shelf price tag labels located on the product storage structure and based on a definition for each of the one or more separate product storage spaces of the product storage structure, determine at least a first price tag label that is allocated to at least a first product storage space of the separate product storage spaces of the product storage structure; and
based on a determination of at least the first price tag label that is allocated to at least the first product storage space, associate at least the first price tag label with each of the one or more products in the group of the one or more identical products stored in the first product storage space.
2. The system of claim 1 , wherein the control circuit is programmed to generate a first set of virtual boundary lines and a second set of virtual boundary lines, wherein each of the virtual boundary lines of the first set surrounds an individual one of the price tag labels detected in the at least one image, and wherein each of the virtual boundary lines of the second set surrounds an individual one of the products detected in the at least one image.
3. The system of claim 2 ,
wherein the product storage structure comprises a plurality of horizontal support members interconnected by vertical support members; and
wherein the control circuit is further programmed to:
analyze the at least one image of the product storage structure to detect individual ones of the vertical support members of the product storage structure; and
based on detection of the individual ones of the vertical support members of the product storage structure, generate a third set of virtual boundary lines, wherein each one of the virtual boundary lines of the third set surrounds an individual one of the vertical support members detected in the at least one image.
4. The system of claim 3 , wherein the virtual boundary lines of the first set, the second set, and the third set are 2-dimensional, and wherein the control circuit is further programmed to:
estimate depth from the horizontal support members of the product storage structure to the image capture device based on actual size of the price tag labels in the at least one image and based on a pixel size of the price tag labels in the at least one image;
project the virtual boundary lines of the first set, the second set, and the third set into a 3-dimensional coordinate system based on the estimated depth;
align points of the 3-dimensional coordinate system based on location information received from the at least one image capture device; and
re-project the aligned points of the 3-dimensional coordinate system back to a 2-dimensional space.
5. The system of claim 4 , wherein the control circuit is further programmed to cluster the virtual boundary lines of at least one of the first, second, and third sets to define each of the separate product storage spaces containing the group of the identical products therein.
6. The system of claim 5 ,
wherein the control circuit is programmed to generate a fourth set of virtual boundary lines, wherein each of the virtual boundary lines of the fourth set surrounds one of the defined separate product storage spaces containing the group of the identical products therein; and
wherein the first product storage space has two or more adjacent separate product storage spaces that are each surrounded by virtual boundary lines of the fourth set that adjoin the virtual boundary lines of the fourth set that surround the first product storage space.
7. The system of claim 5 , wherein the control circuit is programmed to define each of the separate product storage spaces containing the group of the identical products therein based at least on one or both of center cluster coordinates of the price tag labels and center cluster coordinates of the vertical support members.
8. The system of claim 1 , wherein, based on a determination by the control circuit that at least one of the separate product storage spaces does not have a price tag label allocated thereto, the control circuit is programmed to generate a missing label alert and to transmit the missing label alert.
9. The system of claim 1 , further comprising an electronic database configured to store the images captured by the image capture device, and wherein the control circuit is programmed to transmit, to the electronic database for storage, a signal including electronic data associating the first price tag label with each of the products in the group of identical products stored in the first product storage space.
10. A method of analyzing on-shelf price tag labels and products at a product storage facility, the method comprising:
capturing, via an image capture device having a field of view that includes at least a portion of a product storage structure implemented on a motorized robotic unit, one or more images of a product storage structure at the product storage facility having the products arranged thereon, wherein the motorized robotic unit includes wheels permitting the motorized robotic unit to move about the product storage facility;
obtaining, by a computing device communicatively coupled to the image capture device and including a control circuit, at least one image of the product storage structure captured by the image capture device;
by the control circuit of the computing device:
analyzing, using one or more computer vision models communicatively coupled to a control circuit of a computing device, wherein the one or more computer vision models are trained using data sets including at least raw image data collected from the captured one or more images, reference image data associated with reference images of one or more product storage structures, reference images of one or more products, reference images of one or more price tag labels, and planogram data associated with the product storage facility, the at least one image of the product storage structure captured by the image capture device to detect individual ones of the price tag labels located on the product storage structure;
analyzing the at least one image of the product storage structure to detect individual ones of the products located on the product storage structure;
based on detection of the individual ones of the products located on the product storage structure, defining separate product storage spaces of the product storage structure, wherein each of the separate product storage spaces contains a group of identical products therein;
based on detection of the individual ones of the price tag labels located on the product storage structure and based on a definition of the separate product storage spaces of the product storage structure, determining a first price tag label that is allocated to a first product storage space of the separate product storage spaces of the product storage structure; and
based on a determination of the first price tag label that is allocated to the first product storage space, associating the first price tag label with each of the products in the group of identical products stored in the first product storage space.
11. The method of claim 10 , further comprising, generating, by the control circuit, a first set of virtual boundary lines and a second set of virtual boundary lines, wherein each of the virtual boundary lines of the first set surrounds an individual one of the price tag labels detected in the at least one image, and wherein each of the virtual boundary lines of the second set surrounds an individual one of the products detected in the at least one image.
12. The method of claim 11 ,
Wherein the product storage structure comprises a plurality of horizontal support members interconnected by vertical support members; and
further comprising, by the control circuit:
analyzing the at least one image of the product storage structure to detect individual ones of the vertical support members of the product storage structure; and
based on detection of the individual ones of the vertical support members of the product storage structure, generating a third set of virtual boundary lines, wherein each one of the virtual boundary lines of the third set surrounds an individual one of the vertical support members detected in the at least one image.
13. The method of claim 12 , wherein the virtual boundary lines of the first set, the second set, and the third set are 2-dimensional, and further comprising, by the control circuit:
estimating depth from the horizontal support members of the product storage structure to the image capture device based on actual size of the price tag labels in the at least one image and based on pixel size of the price tag labels in the at least one image;
projecting the virtual boundary lines of the first set, the second set, and the third set into a 3-dimensional coordinate system based on the estimated depth;
aligning points of the 3-dimensional coordinate system based on location information received from the at least one image capture device; and
re-projecting the aligned points of the 3-dimensional coordinate system back to a 2-dimensional space.
14. The method of claim 13 , further comprising, by the control circuit, clustering the virtual boundary lines of at least one of the first, second, and third sets to define each of the separate product storage spaces containing the group of the identical products therein.
15. The method of claim 14 ,
further comprising, by the control circuit, generating a fourth set of virtual boundary lines, wherein each of the virtual boundary lines of the fourth set surrounds one of the defined separate product storage spaces containing the group of the identical products therein; and
wherein the first product storage space has two or more adjacent separate product storage spaces that are each surrounded by virtual boundary lines of the fourth set that adjoin the virtual boundary lines of the fourth set that surround the first product storage space.
16. The method of claim 14 , further comprising, by the control circuit, defining each of the separate product storage spaces containing the group of the identical products therein based at least on center cluster coordinates of the price tag labels and center cluster coordinates of the vertical support members.
17. The method of claim 10 , further comprising, by the control circuit:
based on a determination by the control circuit that at least one of the separate product storage spaces does not have a price tag label allocated thereto, generating a missing label alert; and
transmitting the missing label alert to a user device of a worker of the retail store to cause the user device of the worker to display a notification including the missing label alert to the worker.
18. The method of claim 10 , further comprising:
storing the images captured by the image capture device in an electronic database, and
transmitting, via the control circuit, a signal to the electronic database for storage, the signal including electronic data associating the first price tag label with each of the products in the group of identical products stored in the first product storage space.