Automated cropping of images using a machine learning predictor
Example systems and methods may selection of video frames using a machine learning (ML) predictor program are disclosed. The ML predictor program may generate predicted cropping boundaries for any given input image. Training raw images associated with respective sets of training master images indicative of cropping characteristics for the training raw image may be input to the ML predictor, and the ML predictor program trained to predict cropping boundaries for raw image based on expected cropping boundaries associated training master images. At runtime, the trained ML predictor program may be applied to runtime raw images in order to generate respective sets of runtime cropping boundaries corresponding to different cropped versions of the runtime raw image. The runtime raw images may be stored with information indicative of the respective sets of runtime boundaries.
1. A method carried out by a machine learning (ML) predictor program implemented on a computing device and configured for generating predicted cropping characteristics for input images, wherein cropping characteristics for any given input image comprise coordinates of cropping boundaries with respect to the any given input image prior to cropping, the method comprising:
receiving one or more uncropped images by the computing device;
applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images a respective set of runtime cropping characteristics, wherein the respective set of runtime cropping characteristics for each respective uncropped image comprises one or more subsets of cropping coordinates for the respective uncropped image, and wherein each subset corresponds to a different cropped version of the respective uncropped image; and
storing, in non-transitory computer-readable memory, the one or more uncropped images together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images,
wherein, prior to receiving the one or more uncropped images, the ML predictor program has been trained to predict cropping characteristics for each respective training raw image of a plurality of training raw images, based on expected cropping characteristics represented in a respective set of training master images associated with the respective training raw image,
and wherein each training master image of the respective set of training master images indicates respective cropping characteristics defined for the associated respective training raw image.
2. The method of claim 1 , further comprising:
for each respective set of runtime cropping characteristics associated with each respective uncropped image, computing a predicted confidence level of each of the one or more subsets of cropping coordinates; and
storing, in the non-transitory computer-readable memory, the predicted confidence levels of the one or more subsets together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images.
3. The method of claim 1 , wherein storing, in the non-transitory computer-readable memory, the one or more uncropped images together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images comprises storing each respective uncropped image together with at least one of:
metadata corresponding to the respective set of runtime cropping characteristics, wherein the metadata are applicable to the respective uncropped image to create each different cropped version of the respective uncropped image; or
each different cropped version of the respective uncropped image generated by application of the set of runtime cropping characteristics to the respective uncropped image.
4. The method of claim 1 , wherein the ML predictor program comprises an artificial neural network (ANN),
and wherein applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images the respective set of runtime cropping characteristics comprises applying the ANN to the one or more uncropped images to predict the respective set of runtime cropping characteristics for each of the one or more uncropped images.
5. The method of claim 1 , wherein the cropping characteristics of each respective set of training master images define one or more rectangular training bounding boxes, each enclosing a respective region of interest (ROI) of the respective training raw image associated with the respective set of training master image,
and wherein each rectangular training bounding box has a respective fixed aspect ratio specified according to a set of predetermined training aspect ratios.
6. The method of claim 5 , wherein applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images the respective set of runtime cropping characteristics comprises applying the ML predictor program to the one or more uncropped images to predict for each respective runtime raw image one or more respective runtime rectangular bounding boxes.
7. The method of claim 1 , further comprising:
recognizing, by the ML predictor program, an exclusion boundary around any particular image; and
excluding, by the ML predictor program, any portion of the any particular image within the exclusion boundary from consideration in computations to predict the cropping characteristics of the any particular image.
8. The method of claim 1 , wherein the one or more uncropped images comprise digital still images of digital streaming media content,
and wherein each cropped version of the respective uncropped image is configured for display in at least one of promotional communication associated with the streaming media content, or electronic program control of the streaming media content.
9. A system configured for generating predicted cropping characteristics for input images, wherein cropping characteristics for any given input image comprise coordinates of cropping boundaries with respect to the any given input image prior to cropping, the system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the system to carry out operations of a machine learning (ML) predictor program, wherein the operations include:
receiving one or more uncropped images;
applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images a respective set of runtime cropping characteristics, wherein the respective set of runtime cropping characteristics for each respective uncropped image comprises one or more subsets of cropping coordinates for the respective uncropped image, and wherein each subset corresponds to a different cropped version of the respective uncropped image; and
storing, in non-transitory computer-readable memory, the one or more uncropped images together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images,
wherein, prior to receiving the one or more uncropped images, the ML predictor program has been trained to predict cropping characteristics for each respective training raw image of a plurality of training raw images, based on expected cropping characteristics represented in a respective set of training master images associated with the respective training raw image,
and wherein each training master image of the respective set of training master images indicates respective cropping characteristics defined for the associated respective training raw image.
10. The system of claim 9 , wherein the operations further include:
for each respective set of runtime cropping characteristics associated with each respective uncropped image, computing a predicted confidence level of each of the one or more subsets of cropping coordinates; and
storing, in the non-transitory computer-readable memory, the predicted confidence levels of the one or more subsets together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images.
11. The system of claim 9 , wherein storing, in the non-transitory computer-readable memory, the one or more uncropped images together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images comprises storing each respective uncropped image together with at least one of:
metadata corresponding to the respective set of runtime cropping characteristics, wherein the metadata are applicable to the respective uncropped image to create each different cropped version of the respective uncropped image; or
each different cropped version of the respective uncropped image generated by application of the set of runtime cropping characteristics to the respective uncropped image.
12. The system of claim 9 , wherein the ML predictor program comprises an artificial neural network (ANN),
and wherein applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images the respective set of runtime cropping characteristics comprises applying the ANN to the one or more uncropped images to predict the respective set of runtime cropping characteristics for each of the one or more uncropped images.
13. The system of claim 9 , wherein the cropping characteristics of each respective set of training master images define one or more rectangular training bounding boxes, each enclosing a respective region of interest (ROI) of the respective training raw image associated with the respective set of training master image,
and wherein each rectangular training bounding box has a respective fixed aspect ratio specified according to a set of predetermined training aspect ratios.
14. The system of claim 13 , wherein applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images the respective set of runtime cropping characteristics comprises applying the ML predictor program to the one or more uncropped images to predict for each respective runtime raw image one or more respective runtime rectangular bounding boxes.
15. The system of claim 9 , wherein the operations further include:
recognizing, by the ML predictor program, an exclusion boundary around any particular image; and
excluding, by the ML predictor program, any portion of the any particular image within the exclusion boundary from consideration in computations to predict the cropping characteristics of the any particular image.
16. The system of claim 9 , wherein the one or more uncropped images comprise digital still images of digital streaming media content,
and wherein each cropped version of the respective uncropped image is configured for display in at least one of promotional communication associated with the streaming media content, or electronic program control of the streaming media content.
17. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors of a system configured for generating predicted cropping characteristics for input images, wherein cropping characteristics for any given input image comprise coordinates of cropping boundaries with respect to the any given input image prior to cropping, cause the system to carry out operations of a machine learning (ML) predictor program, wherein the operations include:
receiving one or more uncropped images;
applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images a respective set of runtime cropping characteristics, wherein the respective set of runtime cropping characteristics for each respective uncropped image comprises one or more subsets of cropping coordinates for the respective uncropped image, and wherein each subset corresponds to a different cropped version of the respective uncropped image; and
storing, in non-transitory computer-readable memory, the one or more uncropped images together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images,
wherein, prior to receiving the one or more uncropped images, the ML predictor program has been trained to predict cropping characteristics for each respective training raw image of a plurality of training raw images, based on expected cropping characteristics represented in a respective set of training master images associated with the respective training raw image,
and wherein each training master image of the respective set of training master images indicates respective cropping characteristics defined for the associated respective training raw image.
18. The non-transitory computer-readable medium of claim 17 , wherein the operations further include:
for each respective set of runtime cropping characteristics associated with each respective uncropped image, computing a predicted confidence level of each of the one or more subsets of cropping coordinates; and
storing, in the non-transitory computer-readable memory, the predicted confidence levels of the one or more subsets together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images.
19. The non-transitory computer-readable medium of claim 17 , wherein storing, in the non-transitory computer-readable memory, the one or more uncropped images together with the respective set of runtime cropping characteristics for each respective uncropped image of the one or more uncropped images comprises storing each respective uncropped image together with at least one of:
metadata corresponding to the respective set of runtime cropping characteristics, wherein the metadata are applicable to the respective uncropped image to create each different cropped version of the respective uncropped image; or
each different cropped version of the respective uncropped image generated by application of the set of runtime cropping characteristics to the respective uncropped image.
20. The non-transitory computer-readable medium of claim 17 , wherein the ML predictor program comprises an artificial neural network (ANN),
and wherein applying the ML predictor program to the one or more uncropped images in order to generate for each respective uncropped image of the one or more uncropped images the respective set of runtime cropping characteristics comprises applying the ANN to the one or more uncropped images to predict the respective set of runtime cropping characteristics for each of the one or more uncropped images.