Automatically generating an image dataset based on object instance similarity
Methods, systems, and non-transitory computer readable media are disclosed for accurately and efficiently generating groups of images portraying semantically similar objects for utilization in building machine learning models. In particular, the disclosed system utilizes metadata and spatial statistics to extract semantically similar objects from a repository of digital images. In some embodiments, the disclosed system generates color embeddings and content embeddings for the identified objects. The disclosed system can further group similar objects together within a query space by utilizing a clustering algorithm to create object clusters and then refining and combining the object clusters within the query space. In some embodiments, the disclosed system utilizes one or more of the object clusters to build a machine learning model.
1 . A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:
select a series of correlated images from digital images utilizing metadata associated with the digital images;
extract objects portrayed in the series of correlated images by:
detecting the objects in the series of correlated images by utilizing an object detection model;
generating segmentation masks of the objects in the series of correlated images by utilizing a segmentation model; and
removing image data not associated with the segmentation masks from the series of correlated images to generate extracted objects;
generate content embeddings and color embeddings for the extracted objects;
generate object clusters by grouping semantically similar objects from the content embeddings and the color embeddings utilizing a clustering algorithm;
deduplicate the object clusters by merging nearest object clusters of the object clusters; and
utilize one or more of the deduplicated object clusters and corresponding digital images to build a machine learning model by:
selecting a pair of ground truth similar images from an object cluster of the deduplicated object clusters; and
utilizing the pair of ground truth similar images to train a digital image similarity machine learning model.
2 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to select the series of correlated images by:
determining, based on the metadata associated with the digital images, a subset of digital images having at least one of time metadata tags within a threshold time period, location metadata tags within a threshold distance, or matching user identification metadata tags; and
utilizing the subset of digital images to select the series of correlated images.
3 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to select the digital images by:
determining one or more historical image queries or upload keywords;
determining a set of images corresponding to the one or more historical image queries or the upload keywords; and
utilizing the set of images, extracting the digital images utilized to select the series of correlated images.
4 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to process a plurality of digital images utilizing the machine learning model to generate a plurality of digital embeddings for use in responding to digital image queries.
5 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the content embeddings by:
generating feature vectors from the extracted objects by utilizing a convolutional neural network trained to generate semantic image labels; and
utilizing the feature vectors as the content embeddings to generate the object clusters.
6 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the color embeddings by:
grouping pixels of the extracted objects into a plurality of colors to generate color histograms; and
utilizing the color histograms as the color embeddings to generate the object clusters.
7 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the object clusters by:
combining the content embeddings and the color embeddings for the extracted objects; and
mapping the combined embeddings corresponding to the extracted objects in a query space.
8 . The non-transitory computer readable medium as recited in claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the object clusters comprises utilizing a density-based clustering algorithm to generate the object clusters from the combined embeddings corresponding to the extracted objects in the query space.
9 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to deduplicate the object clusters by:
generating a mean cluster embedding for an object cluster of the object clusters;
generating a nearest mean cluster embedding for a nearest object cluster; and
merging the nearest object cluster with the object cluster based on a distance between the nearest mean cluster embedding and the mean cluster embedding falling within a threshold distance value.
10 . The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to utilize the one or more of the object clusters to build the machine learning model by utilizing groups of digital images corresponding with the one or more of the object clusters as a ground truth similarity data set for building the digital image similarity machine learning model.
11 . A system comprising:
at least one memory device comprising digital images having metadata; and
at least one server device configured to cause the system to:
select a series of correlated images from the digital images utilizing the metadata associated with the digital images by grouping digital images having related metadata tags;
extract objects portrayed in the series of correlated images by generating background masks for the series of correlated images utilizing an object detection model and a segmentation model;
generate content embeddings and color embeddings for the extracted objects by utilizing a convolutional neural network trained to generate semantic image labels and a color embedding model;
generate object clusters by grouping semantically similar objects from the content embeddings and the color embeddings utilizing a clustering algorithm by mapping combined embeddings in a query space; and
deduplicate the object clusters by merging nearest object clusters of the object clusters based on a threshold distance value; and
build a digital image similarity machine learning model by:
selecting a pair of ground truth similar images from an object cluster of the deduplicated object clusters; and
utilizing the pair of ground truth similar images to train the digital image similarity machine learning model.
12 . The system as recited in claim 11 , wherein the at least one server device is further configured to cause the system to generate the extracted objects by removing image data not associated with the background masks from the series of correlated images.
13 . The system as recited in claim 11 , wherein the at least one server device is further configured to cause the system to utilize the digital image similarity machine learning model by processing a plurality of digital images utilizing the digital image similarity machine learning model to generate a plurality of digital image embeddings for use in responding to digital image queries.
14 . The system as recited in claim 11 , wherein the at least one server device is further configured to cause the system to:
utilize one or more of the deduplicated object clusters and instance labels associated with the one or more of the deduplicated object clusters to build an object classification machine learning model; and
utilize the object classification machine learning model to generate predicted instance labels for objects portrayed in a plurality of images.
15 . The system as recited in claim 11 , wherein the at least one server device is further configured to generate the background masks for the series of correlated images by:
detecting the objects in the series of correlated images by utilizing the object detection model;
generating segmentation masks of the objects in the series of correlated images by utilizing the segmentation model; and
generating the background masks by inverting the segmentation masks.
16 . In a digital medium environment for processing a repository of digital images, a computer-implemented method comprising:
select a series of correlated images from digital images utilizing metadata associated with the digital images;
extracting objects portrayed in a series of correlated images by:
detecting the objects in the series of correlated images by utilizing an object detection model;
generating segmentation masks of the objects in the series of correlated images by utilizing a segmentation model; and
removing image data not associated with the segmentation masks from the series of correlated images to generate extracted objects;
generating content embeddings and color embeddings for the extracted objects;
generating object clusters by grouping the content embeddings and the color embeddings utilizing a clustering algorithm;
utilizing one or more of the object clusters and corresponding digital images to build a digital image similarity machine learning model;
deduplicating the object clusters by merging nearest object clusters of the object clusters based on a threshold distance value; and
utilizing one or more of the deduplicated object clusters to build the digital image similarity machine learning model by:
selecting a pair of ground truth similar images from an object cluster of the deduplicated object clusters; and
utilizing the pair of ground truth images to train the digital image similarity machine learning model.
17 . The computer-implemented method as recited in claim 16 , further comprising generating the content embeddings by:
generating feature vectors from the extracted objects by utilizing a convolutional neural network trained to generate semantic image labels; and
utilizing the feature vectors as the content embeddings to generate the object clusters.
18 . The computer-implemented method as recited in claim 16 , further comprising generating the color embeddings by generating color histograms from the extracted objects.
19 . The computer-implemented method as recited in claim 16 , further comprising:
processing a plurality of digital images utilizing the digital image similarity machine learning model to generate a plurality of digital image embeddings for use in responding to digital image queries.
20 . The computer-implemented method as recited in claim 16 , further comprising generating the object clusters by:
combining the content embeddings and the color embeddings for the extracted objects; and
mapping the combined embeddings corresponding to the extracted objects in a query space.