Restricted multi-scale inference for machine learning
Techniques for utilizing multiple scales of images as input to machine learning (ML) models are discussed herein. Operations can include providing an image associated with a first scale to a first ML model. An output of the first ML model can include a first bounding box indicative of a first region of the image representing a first object, with the first bounding box falling within a first range of sizes. Next, a scaled image can be generated by scaling the image. The scaled image can be provided to a second ML model, which can output a second bounding box indicative of a second region of the image representing a second object, the second bounding falling within a second range of sizes. Thus, inputting a scaled image to a same ML model (or to different ML models) can result in different detected features in the images.
1. A system comprising:
one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving an image having a first scale;
providing the image to a first machine-learning (ML) model trained to output regions of interest (ROIs) having sizes within a first range of sizes,
wherein the first ML model is trained to output the ROIs having the sizes within the first range of sizes based at least in part on:
training the first ML model, wherein the training comprises:
providing, as input to the first ML model, test images that include test objects associated with areas defined by reference regions; and
determining, by the first ML model and based at least in part on the test images, multiple ROIs;
determining degrees of alignment of the multiple ROIs to an area of the areas defined by the reference regions; and
determining the first range of sizes based at least in part on identifying a span of object sizes that corresponds to a portion of the degrees of alignment that meet or exceed a threshold degree of alignment;
receiving, from the first ML model, a first ROI indicative of a first region of the image representing at least part of a first object, the first ROI having a first size within the first range of sizes;
scaling, based at least in part on a second scale, the image as a scaled image, the scaled image and the image being different sizes;
providing the scaled image to a second ML model trained to output ROIs having sizes within a second range of sizes, wherein the first range of sizes and the second range of sizes are different from each other; and
receiving, from the second ML model, a second ROI indicative of a second region of the image representing at least part of a second object, the second ROI having a second size within the second range of sizes.
2. The system as claim 1 recites, wherein the image is provided to the first ML model and the scaled image is provided to the second ML model substantially simultaneously.
3. The system as claim 1 recites, wherein the first ML model and the second ML model are a same ML model.
4. The system as claim 1 recites, the operations further comprising:
determining the first range of sizes for the first ML model based at least in part on:
providing, as input to the first ML model, a first batch of training images that include objects of different sizes;
determining, based at least in part on ground truths associated with the objects, accuracies of the first ML model as a function of ROI sizes; and
determining the first range of sizes based at least in part on identifying a range of ROI sizes that correspond to accuracies that meet or exceed a threshold accuracy.
5. A computer-implemented method comprising:
receiving a first scaled image and a second scaled image, the first scaled image and the second scaled image being two differently sized versions of a same image;
providing, as first input to a first machine-learning (ML) model trained to output regions of interest (ROIs) having sizes within a first range of sizes, the first scaled image,
wherein the first ML model is trained to output the ROIs having the sizes within the first range of sizes based at least in part on:
training the first ML model, wherein the training comprises:
providing, as input to the first ML model, test images that include test objects associated with areas defined by reference regions; and
determining, by the first ML model and based at least in part on the test images, multiple ROIs;
determining degrees of alignment of the multiple ROIs to an area of the areas defined by the reference regions; and
determining the first range of sizes based at least in part on identifying a span of object sizes that corresponds to a portion of the degrees of alignment that meet or exceed a threshold degree of alignment;
receiving, as first output from the first ML model, one or more first ROIs having sizes within the first range of sizes;
providing, as second input to a second ML model trained to output ROIs having sizes within a second range of sizes, wherein the first range of sizes and the second range of sizes are different from each other, the second scaled image; and
receiving, as second output from the second ML model, one or more second ROIs having sizes within the second range of sizes.
6. The computer-implemented method as claim 5 recites, wherein:
the image includes representations of a first object and a second object, and
the computer-implemented method further comprises:
determining, by the first ML model and based at least in part on the first scaled image, a first ROI that represents the first object and a second ROI that represents the second object; and
determining, by the second ML model and based at least in part on the second scaled image, a third ROI that represents the first object and a fourth ROI that represents the second object.
7. The computer-implemented method as claim 6 recites, wherein:
the first ROI has a first size outside the first range of sizes;
the second ROI has a second size within the first range of sizes;
the third ROI has a third size outside the second range of sizes;
the fourth ROI has a fourth size within the second range of sizes;
the first ML model discards the first ROI and outputs the second ROI; and
the second ML model discards the fourth ROI and outputs the third ROI.
8. The computer-implemented method as claim 5 recites, further comprising:
scaling, based at least in part on a first scale, the image to generate the first scaled image; and
scaling, based at least in part on a second scale, the image to generate the second scaled image.
9. The computer-implemented method as claim 5 recites, wherein the first scaled image is provided to the first ML model and the second scaled image is provided to the second ML model substantially simultaneously.
10. The computer-implemented method as claim 5 recites, further comprising determining the first range of sizes based at least in part on an accuracy associated with a first ROI of the one or more first ROIs, the determining comprising:
determining, based at least in part on a ground truth associated with an area of the image representing an object, a degree of alignment of the first ROI to the area defined by the ground truth; and
determining the first range of sizes based at least in part on identifying a range of ROI sizes that corresponds to degrees of alignment determined for the one or more first ROIs that meet or exceed a threshold degree of alignment.
11. The computer-implemented method as claim 5 recites, further comprising:
receiving a batch of images, wherein the batch of images includes a first predefined number of images that are associated with a first object classification and a second predefined number of images that are associated with a second object classification; and
training the first ML model based at least in part on providing the batch of images as input to the first ML model,
wherein the first predefined number of images and the second predefined number of images as based at least in part on a confidence score associated with the first ML model or the second ML model.
12. A non-transitory computer-readable medium having a set of instructions that, when executed, cause one or more processors to perform operations comprising:
receiving a first scaled image and a second scaled image, the first scaled image and the second scaled image corresponding to an image;
providing, as first input to a first machine-learning (ML) model trained to output regions of interest (ROIs) having sizes within a first range of sizes, the first scaled image,
wherein the first ML model is trained to output the ROIs having the sizes within the first range of sizes based at least in part on:
training the first ML model, wherein the training comprises:
providing, as input to the first ML model, test images that include test objects associated with areas defined by reference regions; and
determining, by the first ML model and based at least in part on the test images, multiple ROIs;
determining degrees of alignment of the multiple ROIs to an area of the areas defined by the reference regions; and
determining the first range of sizes based at least in part on identifying a span of object sizes that corresponds to a portion of the degrees of alignment that meet or exceed a threshold degree of alignment;
receiving, as first output from the first ML model, one or more first ROIs having sizes within the first range of sizes;
providing, as second input to a second ML model trained to output ROIs having sizes within a second range of sizes, wherein the first range of sizes and the second range of sizes are different from each other, the second scaled image; and
receiving, as second output from the second ML model, one or more second ROIs having sizes within the second range of sizes.
13. The non-transitory computer-readable medium as claim 12 recites, wherein:
the image includes representations of a first object and a second object, and
the operations further comprise:
determining, by the first ML model and based at least in part on the first scaled image, a first ROI that represents the first object and a second ROI that represents the second object; and
determining, by the second ML model and based at least in part on the second scaled image, a third ROI that represents the first object and a fourth ROI that represents the second object.
14. The non-transitory computer-readable medium as claim 13 recites, wherein:
the first ROI has a first size outside the first range of sizes;
the second ROI has a second size within the first range of sizes;
the third ROI has a third size outside the second range of sizes;
the fourth ROI has a fourth size within the second range of sizes;
the first ML model discards the first ROI and outputs the second ROI; and
the second ML model discards the fourth ROI and outputs the third ROI.
15. The non-transitory computer-readable medium as claim 12 recites, wherein the operations further comprise:
scaling, based at least in part on a first scale, the image to generate the first scaled image; and
scaling, based at least in part on a second scale, the image to generate the second scaled image.
16. The non-transitory computer-readable medium as claim 12 recites, wherein the first scaled image is provided to the first ML model and the second scaled image is provided to the second ML model substantially simultaneously.
17. The non-transitory computer-readable medium as claim 12 recites, wherein the operations further comprise determining the first range of sizes based at least in part on an accuracy associated with a first ROI of the one or more first ROIs, the determining comprising:
determining, based at least in part on a ground truth associated with an area of the image representing an object, a degree of alignment of the first ROI to the area defined by the ground truth; and
determining the first range of sizes based at least in part on identifying a range of ROI sizes that corresponds to degrees of alignment determined for the one or more first ROIs that meet or exceed a threshold degree of alignment.
18. The non-transitory computer-readable medium as claim 12 recites, wherein the operations further comprise:
receiving a batch of images, wherein the batch of images includes a first predefined number of images that are associated with a first object classification and a second predefined number of images that are associated with a second object classification; and
training the first ML model based at least in part on providing the batch of images as input to the first ML model,
wherein the first predefined number of images and the second predefined number of images as based at least in part on a confidence score associated with the first ML model or the second ML model.
19. The non-transitory computer-readable medium as claim 12 recites, wherein the operations further comprising determining to associate an ROI with a representation of an object in the image, wherein determining to associate the ROI with the representation is based at least in part on determining to use the first ROIs or the second ROIs based at least in part on a dimension of the representation in the image, the first range of sizes, and the second range of sizes.