Systems and methods of updating model templates associated with images of retail products at product storage facilities
Systems and methods of updating templates for use in recognizing individual products in images captured at a product storage facility include an image capture device that captures one or more images of product storage structure at a product storage facility, a computing device in communication with the image capture device, and an electronic database that stores keyword model templates and feature model templates associated with images of previously recognized individual products detected at the product storage facility. The computing device obtains the keyword and feature model templates associated with a recognized product from the electronic database, extracts the keywords from the products associated with the obtained keyword model templates, identifies products that are similar to the recognized product, and updates the keyword model template for each of the products to include must keywords and negative keywords, facilitating recognition of products in subsequent images captured by the image capture device.
1. A system of updating templates for use in recognizing individual products in images captured at a product storage facility, the system comprising:
an image capture device having a field of view that includes at least a portion of a product storage structure of the product storage facility, the product storage structure having the individual products arranged thereon, wherein the image capture device is configured to capture one or more images of the product storage structure;
a computing device including a control circuit, the computing device being communicatively coupled to the image capture device; and
an electronic database configured to store keyword model templates and feature model templates associated with the images of previously recognized individual products stored at the product storage facility, wherein the keyword model templates includes an image of a recognized individual product and meta data associated with the recognized individual product, and wherein the feature model templates include the image of the recognized individual product in association with visual features of the recognized individual product;
wherein the control circuit of the computing device is configured to:
obtain at least one of the keyword model templates and feature model templates stored in the electronic database;
extract one or more keywords from each recognized individual product depicted in captured images associated with the obtained at least one of the keyword model templates;
correlate the one or more keywords extracted from each recognized individual product depicted in the images associated with the at least one of the keyword model templates to identify similar products, where the similar products share a number of keywords with each other and do not share a number of keywords with each other;
update a keyword model template for each of the similar products to:
set keywords that are unique to the similar products as must keywords; and
set the must keywords that are not shared between the similar products as negative keywords;
transmit the updated keyword model template including the must keywords and the negative keywords for each of the similar products to the electronic database for storage to be used for analysis of subsequent images captured by the image capture device, and recognition of products in the subsequent images;
obtain images of the product storage structure captured by the image capture device;
analyze the obtained images to detect individual ones of the individual products located on the product storage structure;
based on detection of the individual ones of the individual products in the obtained images, recognize the individual ones of the individual products detected in the obtained images as corresponding to a known product identifier;
crop each of the individual ones of the recognized individual ones of the individual products from the images to generate a plurality of cropped images;
create a plurality of image clusters using the plurality of cropped images, wherein each of the plurality of image clusters contains cropped images that depict an identical one of the recognized individual ones of the individual products corresponding to the known product identifier;
select one of the cropped images in each of the plurality of image clusters as a centroid image, wherein the centroid image is designated as the updated keyword model template representing the identical one of the recognized individual ones of the individual products;
resample a number of the cropped images of each of the plurality of clusters that are located closest to the centroid image by virtue of having embeddings that are most similar to embeddings of the centroid image; and
select the centroid image and the resampled number of cropped images as an updated feature model template representing the one of the recognized individual ones of the individual products.
2. The system of claim 1 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage structure.
3. The system of claim 1 , wherein the control circuit sets keywords that are shared between the similar products as high frequency keywords.
4. The system of claim 1 , wherein the keyword model templates and the feature model templates are generated based on processing, by the control circuit of the computing device, of a first batch of the images of the product storage structure from the obtained images to crop the products detected in the first batch of the images to generate a first batch of cropped images, each of the cropped images of the first batch depicting an individual one of the products detected in the first batch.
5. The system of claim 4 , wherein:
the obtained images further include a second batch of the images of the product storage structure captured by the image capture device; and
cropped images of the first batch that correspond to the known product identifier and cropped images of the second batch are combined to create the plurality of image clusters.
6. The system of claim 5 , wherein the control circuit is programmed to:
generate embeddings for each of the cropped images of the second batch, wherein the embeddings represent dense vector representations of the cropped images of the second batch;
determine a similarly of the embeddings between the cropped images of the second batch in each of the plurality of image clusters; and
position the cropped images of the second batch in each of the plurality of image clusters based on a similarity between the embeddings of the cropped images of the second batch in each of the plurality of image clusters.
7. The system of claim 6 ,
wherein the control circuit is programmed to perform optical character recognition to detect one or more keywords from each of the individual products depicted in the cropped images of the second batch; and
wherein the updated keyword model template includes additional keywords detected via the optical character recognition in at least one of the cropped images of the second batch.
8. The system of claim 6 ,
wherein the control circuit is programmed to correlate embeddings of the cropped images of the first batch used to generate the keyword model templates and the feature model templates stored in the electronic database with the embeddings generated by the control circuit for the cropped images of the second batch; and
wherein the updated feature model template includes at least some of the embeddings associated with the cropped images of the second batch.
9. The system of claim 6 ,
wherein the control circuit is programmed to remove at least one of the cropped images of the first batch from one of the feature model templates previously stored in the electronic database based on a determination that the at least one of the cropped images of the first batch was incorrectly associated with the one of the feature model templates previously stored in the electronic database; and
wherein the control circuit is programmed to remove at least one of the cropped images of the second batch from the updated feature model template based on a determination that the at least one of the cropped images of the second batch was incorrectly associated with the updated feature model template.
10. A method of updating templates for use in recognizing individual products in images captured at a product storage facility, the method comprising:
capturing one or more images of a product storage structure of the product storage facility by an image capture device having a field of view that includes at least a portion of the product storage structure, the product storage structure having the individual products arranged thereon;
storing, in an electronic database, keyword model templates and feature model templates associated with the images of previously recognized individual products stored at the product storage facility, wherein the keyword model templates includes an image of a recognized individual product and meta data associated with the recognized individual product, and wherein the feature model templates include the image of the recognized individual product in association with visual features of the recognized individual product;
by a computing device including a control circuit and communicatively coupled to the image capture device:
obtaining at least one of the keyword model templates and feature model templates stored in the electronic database;
extracting one or more keywords from each recognized individual product depicted in captured images associated with the obtained at least one of the keyword model templates;
correlating the one or more keywords extracted from each recognized individual product depicted in the images associated with the at least one of the keyword model templates to identify similar products, where the similar products share a number of keywords with each other and do not share a number of keywords with each other;
updating a keyword model template for each of the similar products to:
set keywords that are unique to the similar products as must keywords; and
set the must keywords that are not shared between the similar products as negative keywords;
transmitting the updated keyword model template including the must keywords and the negative keywords for each of the similar products to the electronic database for storage to be used for analysis of subsequent images captured by the image capture device, and recognition of products in the subsequent images;
obtaining images of the product storage structure captured by the image capture device;
analyzing the obtained images to detect individual ones of the individual products located on the product storage structure;
based on detection of the individual ones of the individual products in the obtained images, recognizing the individual ones of the individual products detected in the obtained images as corresponding to a known product identifier;
cropping each of the individual ones of the recognized individual ones of the individual products from the images to generate a plurality of cropped images;
creating a plurality of image clusters using the plurality of cropped images, wherein each of the plurality of image clusters contains cropped images that depict an identical one of the recognized individual ones of the individual products corresponding to the known product identifier;
selecting one of the cropped images in each of the plurality of image clusters as a centroid image, wherein the centroid image is designated as the updated keyword model template representing the identical one of the recognized individual ones of the individual products;
resampling a number of the cropped images of each of the plurality of clusters that are located closest to the centroid image by virtue of having embeddings that are most similar to embeddings of the centroid image; and
selecting the centroid image and the resampled number of cropped images as an updated feature model template representing the one of the recognized individual ones of the individual products.
11. The method of claim 10 , wherein the image capture device comprises a motorized robotic unit that includes wheels that permit the motorized robotic unit to move about the product storage facility, and a camera to permit the motorized robotic unit to capture the one or more images of the product storage structure.
12. The method of claim 10 , further comprising, by the control circuit, setting keywords that are shared between the similar products as high frequency keywords.
13. The method of claim 10 , wherein the keyword model templates and the feature model templates are generated based on processing, by the control circuit of the computing device, of a first batch of the images of the product storage structure from the obtained images to crop the products detected in the first batch of the images to generate a first batch of cropped images, each of the cropped images of the first batch depicting an individual one of the products detected in the first batch.
14. The method of claim 13 , wherein:
the obtained images further include a second batch of the images of the product storage structure captured by the image capture device; and
cropped images of the first batch that correspond to the known product identifier and cropped images of the second batch are combined to create the plurality of image clusters.
15. The method of claim 14 , further comprising, by the control circuit:
generating embeddings for each of the cropped images of the second batch, wherein the embeddings represent dense vector representations of the cropped images of the second batch;
determining a similarly of the embeddings between the cropped images of the second batch in each of the plurality of image clusters; and
positioning the cropped images of the second batch in each of the plurality of image clusters based on a similarity between the embeddings of the cropped images of the second batch in each of the plurality of image clusters.
16. The method of claim 15 , further comprising, performing, by the control circuit, optical character recognition to detect one or more keywords from each of the individual products depicted in the cropped images of the second batch, wherein the updated keyword model template includes additional keywords detected via the optical character recognition in at least one of the cropped images of the second batch.
17. The method of claim 15 , further comprising, by the control circuit, correlating embeddings of the cropped images of the first batch used to generate the keyword model templates and the feature model templates stored in the electronic database with the embeddings generated by the control circuit for the cropped images of the second batch, wherein the updated feature model template includes at least some of the embeddings associated with the cropped images of the second batch.
18. The method of claim 15 , further comprising, by the control circuit:
removing at least one of the cropped images of the first batch from one of the feature model templates previously stored in the electronic database based on a determination that the at least one of the cropped images of the first batch was incorrectly associated with the one of the feature model template previously stored in the electronic database; and
removing at least one of the cropped images of the second batch from the updated feature model template based on a determination that the at least one of the cropped images of the second batch of images was incorrectly associated with the updated feature model template.