System and method for iterative refinement and curation of images driven by visual templates
View Patent ↗A method and system for curating images containing specific objects specified by visual templates are described. A set of visual image templates representing an input object is provided to a pre-trained deep neural network (DNN). A feature extraction module is configured to extract a set of feature vectors representing a set of visual image templates and a list of images stored in a DNN-based feature database. A patch generation module is configured to generate image patches representing the input object, from a set of relevant neighbor images providing probable regions where the visual image templates are getting matched. The patch generation module is further configured to train a new network model in a self-supervised manner, with image patches generation by the patch generation module, in an iterative manner.
1 . A computing system, comprising:
a memory; and
a processor coupled to the memory and configured to:
receive a set of visual image templates representing an input object;
receive a plurality of images;
generate a curated subset of images from the plurality of images, each image in the curated subset of images containing the input object, the curated subset of images being generated in an iterative loop, each iteration of the iterative loop including:
extracting a set of feature vectors representing the set of visual image templates by inputting the set of visual image templates into a pre-trained deep neural network (DNN);
extracting a plurality of feature vectors representing the plurality of images by inputting the plurality of images into the pre-trained DNN;
retrieving, based on the set of feature vectors representing the set of visual image templates and the plurality of feature vectors representing the plurality of images, a plurality of relevant neighbor images from the plurality of images representing each of the visual image templates in the set of visual image templates;
generating a plurality of image patches representing the input object from the plurality of relevant neighbor images, each image patch including a region of one of the plurality of relevant neighbor images where the visual image templates are getting matched; and
training a new network model in a self-supervised manner using the plurality of image patches,
wherein the new network model replaces the pre-trained DNN in each subsequent iteration of the iterative loop.
2 . The computing system as claimed in claim 1 , wherein the new network model is provided with the image patches without any labels.
3 . The computing system as claimed in claim 1 , the processor being further configured to filter out false positives from the plurality of image patches.
4 . The computing system as claimed in claim 1 , wherein the set of visual image templates comprises different objects to diversify a range of shapes and content to represent object type of broad object level.
5 . The computing system as claimed in claim 1 , the processor being further configured to compare the set of feature vectors representing the set of visual image templates with the plurality of feature vectors representing the plurality of images in order to retrieve the plurality of relevant neighbor images.
6 . The computing system as claimed in claim 5 , the processor being further configured to compute a similarity score of each of the set of feature vectors with each of the plurality of feature vectors by comparing each of the set of feature vectors with each of the plurality of feature vectors.
7 . A method for curating images containing specific objects specified by visual templates, the method comprising:
receiving, with a processor of a computing system, a set of visual image templates representing an input object;
receiving, with the processor, a plurality of images;
generating, with the processor, a curated subset of images from the plurality of images, each image in the curated subset of images containing the input object, the curated subset of images being generated in an iterative loop, each iteration of the iterative loop including:
extracting a set of feature vectors representing the set of visual image templates by inputting the set of visual image templates into a pre-trained deep neural network (DNN);
extracting a plurality of feature vectors representing the plurality of images by inputting the plurality of images into the pre-trained DNN;
retrieving, based on the set of feature vectors representing the set of visual image templates and the plurality of feature vectors representing the plurality of images, a plurality of relevant neighbor images from the plurality of images representing each of the visual image templates in the set of visual image templates;
generating a plurality of image patches representing the input object from the plurality of relevant neighbor images, each image patch including a region of one of the plurality of relevant neighbor images where the visual image templates are getting matched; and
training a new network model in a self-supervised manner using the plurality of image patches,
wherein the new network model replaces the pre-trained DNN in each subsequent iteration of the iterative loop.