Multi-scale model ensemble for classification of objects
Provided is a multi-scale model ensemble for detection of objects in images. The model ensemble can be applied, for example, in the context of performing object identification activities, such as positively identifying desired objects in image data or video data using a variety of different crop levels.
1 . A method for detecting an object in an image, the method comprising:
receiving, by a computing system comprising at least one electronic processor, an input image;
generating, by the computing system, an image set from the input image, wherein the image set comprises a plurality of images that depict different portions of the input image, at least one of the plurality of images being a cropped image generated based on cropping the input image to depict a cropped portion of the input image;
respectively processing, by the computing system, the plurality of images with a plurality of machine-learned models to generate a plurality of outputs respectively from the plurality of machine-learned models, the cropped image being processed by a first machine-learned model of the plurality of machine-learned models and at least one other image of the plurality of images being processed by a second machine-learned model of the plurality of machine-learned models;
determining, by the computing system, an object detection prediction for the input image based at least in part on the plurality of outputs respectively from the plurality of machine-learned models; and
performing, by the computing system, at least one action based at least in part on the object detection prediction for the input image.
2 . The method of claim 1 , wherein:
the input image depicts a person;
the plurality of images of the image set comprise:
a head image that depicts both a head of the person and a background area that surrounds the head of the person; and
a face image that has been cropped to depict only a face of the person; and
the plurality of machine-learned models comprise:
a machine-learned head model configured to process the head image to generate a head output indicative of whether the input image comprises a particular object; and
a machine-learned face model configured to process the face image to generate a face output indicative of whether the input image comprises the particular object.
3 . The method of claim 1 , wherein:
the input image depicts a person;
the plurality of images of the image set comprise:
a head image that depicts both a head of the person and a background area that surrounds the head of the person; and
one or more image patches that have been cropped to depict only one or more subsets of a face of the person; and
the plurality of machine-learned models comprise:
a machine-learned head model configured to process the head image to generate a head output indicative of whether the input image comprises a particular object; and
a machine-learned patch model configured to process the one or more image patches to generate one or more patch outputs indicative of whether the input image comprises the particular object.
4 . The method of claim 1 , wherein:
the input image depicts a person;
the plurality of images of the image set comprise:
a face image that has been cropped to depict only a face of the person; and
one or more image patches that have been cropped to depict only one or more subsets of the face of the person; and
the plurality of machine-learned models comprise:
a machine-learned face model configured to process the face image to generate a face output indicative of whether the input image comprises a particular object; and
a machine-learned patch model configured to process the one or more image patches to generate one or more patch outputs indicative of whether the input image comprises the particular object.
5 . The method of claim 1 , wherein:
the input image depicts a person;
the plurality of images of the image set comprise:
a head image that depicts both a head of the person and a background area that surrounds the head of the person;
a face image that has been cropped to depict only a face of the person; and
one or more image patches that have been cropped to depict only one or more subsets of the face of the person; and
the plurality of machine-learned models comprise:
a machine-learned head model configured to process the head image to generate a head output indicative of whether the input image comprises a particular object;
a machine-learned face model configured to process the face image to generate a face output indicative of whether the input image comprises the particular object; and
a machine-learned patch model configured to process the one or more image patches to generate one or more patch outputs indicative of whether the input image comprises the particular object.
6 . The method of claim 1 , wherein the plurality of machine-learned models comprise:
a machine-learned pixel-wise model configured to process at least a portion of the input image to generate a grayscale image that indicates a prediction on a pixel-wise basis.
7 . The method of claim 1 , wherein:
the plurality of outputs respectively from the plurality of machine-learned models comprise a plurality of probabilities each indicating a probability that the input imagery comprises the particular object; and
determining, by the computing system, the object detection prediction for the input image based at least in part on the plurality of outputs comprises determining, by the computing system, a weighted average of the plurality of probabilities.
8 . The method of claim 1 , wherein the plurality of machine-learned models have been trained independently of each other.
9 . The method of any of claim 1 , wherein:
the plurality of outputs respectively from the plurality of machine-learned models comprise a plurality of embeddings respectively output by the plurality of machine-learned models; and
determining, by the computing system, the object detection prediction for the input image based at least in part on the plurality of outputs comprises processing, by the computing system, the plurality of embeddings with a machine-learned classification model to generate the object detection prediction as an output of the machine-learned classification model.
10 . The method of any of claim 1 , wherein the plurality of machine-learned models have been jointly trained with each other in an end-to-end fashion.
11 . The method of claim 1 , wherein generating, by the computing system, the image set from the input image comprises:
performing, by the computing system, one or more cropping operations on the input image to generate a plurality of cropped images that depict different portions of the input image; and
performing, by the computing system, one or more scaling operations on the plurality of cropped images to generate the plurality of images of the image set.
12 . The method of claim 1 , wherein:
the input image depicts a person; and
performing, by the computing system, the at least one action based at least in part on the object detection prediction for the input image comprises determining, by the computing system, whether to authenticate the person as an authenticated user based at least in part on the object detection prediction.
13 . The method of claim 1 , wherein the plurality of images included in the image set comprise images having different image scales.
14 . A computer system for detecting an object, the computer system comprising:
at least one electronic processor;
one or more non-transitory computer-readable media that collectively store:
a multi-scale model ensemble comprising a plurality of machine-learned models; and
instructions that, when executed by the computing system, cause the computer system to perform operations, the operations comprising:
receiving an input image;
generating an image set from the input image, wherein the image set comprises a plurality of images that depict different portions of the input image, at least one of the plurality of images being a cropped image generated based on cropping the input image to depict a cropped portion of the input image;
respectively processing the plurality of images with the plurality of machine-learned models to generate a plurality of outputs respectively from the plurality of machine-learned models, the cropped image being processed by a first machine-learned model of the plurality of machine-learned models and at least one other image of the plurality of images being processed by a second machine-learned model of the plurality of machine-learned models;
determining an object detection prediction for the input image based at least in part on the plurality of outputs respectively from the plurality of machine-learned models; and
performing at least one action based at least in part on the object detection prediction for the input image.
15 . The computer system of claim 14 , wherein:
the input image depicts a person;
the plurality of images of the image set comprise two or more of:
a head image that depicts both a head of the person and a background area that surrounds the head of the person;
a face image that has been cropped to depict only a face of the person; and
one or more image patches that have been cropped to depict only one or more subsets of the face of the person; and
the plurality of machine-learned models comprise two or more of:
a machine-learned head model configured to process the head image to generate a head output indicative of whether the input image comprises a particular object;
a machine-learned face model configured to process the face image to generate a face output indicative of whether the input image comprises the particular object; and
a machine-learned patch model configured to process the one or more image patches to generate one or more patch outputs indicative of whether the input image comprises the particular object.
16 . The computer system of claim 14 , wherein the plurality of machine-learned models comprise:
a machine-learned pixel-wise model configured to process at least a portion of the input image to generate a grayscale image that indicates a prediction on a pixel-wise basis.
17 . The computer system of claim 14 , wherein:
the plurality of outputs respectively from the plurality of machine-learned models comprise a plurality of probabilities each indicating a probability that the input imagery comprises the particular object; and
determining the object detection prediction for the input image based at least in part on the plurality of outputs comprises determining a weighted average of the plurality of probabilities.
18 . The computer system of claim 14 , wherein:
the plurality of outputs respectively from the plurality of machine-learned models comprise a plurality of embeddings respectively output by the plurality of machine-learned models; and
determining the object detection prediction for the input image based at least in part on the plurality of outputs comprises processing the plurality of embeddings with a machine-learned classification model to generate the object detection prediction as an output of the machine-learned classification model.
19 . A user device for detecting an object, the user device comprising:
at least one electronic processor;
one or more non-transitory computer-readable media that collectively store:
a multi-scale model ensemble comprising a plurality of machine-learned models; and
instructions that, when executed by the computing system cause the computer system to perform operations, the operations comprising:
receiving an input image;
generating an image set from the input image, wherein the image set comprises a plurality of images that depict different portions of the input image, at least one of the plurality of images being a cropped image generated based on cropping the input image to depict a cropped portion of the input image; and
respectively processing the plurality of images with the plurality of machine-learned models to generate a plurality of outputs respectively from the plurality of machine-learned models, the cropped image being processed by a first machine-learned model of the plurality of machine-learned models and at least one other image of the plurality of images being processed by a second machine-learned model of the plurality of machine-learned models;
determining an object detection prediction for the input image based at least in part on the plurality of outputs respectively from the plurality of machine-learned models; and
performing at least one action based at least in part on the object detection prediction for the input image.
20 . The user device of claim 19 , wherein:
the input image depicts a person;
the plurality of images of the image set comprise two or more of:
a head image that depicts both a head of the person and a background area that surrounds the head of the person;
a face image that has been cropped to depict only a face of the person; and
one or more image patches that have been cropped to depict only one or more subsets of the face of the person; and
the plurality of machine-learned models comprise two or more of:
a machine-learned head model configured to process the head image to generate a head output indicative of whether the input image comprises a particular object;
a machine-learned face model configured to process the face image to generate a face output indicative of whether the input image comprises the particular object; and
a machine-learned patch model configured to process the one or more image patches to generate one or more patch outputs indicative of whether the input image comprises the particular object.