Selection of video frames using a machine learning predictor
Example systems and methods of selection of video frames using a machine learning (ML) predictor program are disclosed. The ML predictor program may generate predicted cropping boundaries for any given input image. Training raw images associated with respective sets of training master images indicative of cropping characteristics for the training raw image may be input to the ML predictor, and the ML predictor program trained to predict cropping boundaries for raw image based on expected cropping boundaries associated training master images. At runtime, the trained ML predictor program may be applied to a sequence of video image frames to determine for each respective video image frame a respective score corresponding to a highest statistical confidence associated with one or more subsets of cropping boundaries predicted for the respective video image frame. Information indicative of the respective video image frame having the highest score may be stored or recorded.
1. A method comprising:
inputting to a machine learning (ML) predictor program implemented on a computing device a first plurality of training raw images, each respective training raw image of the first plurality being associated with a respective set of training master images, each training master image of a given respective set of training master images indicating respective pre-defined cropping characteristics for the associated respective training raw image, wherein the ML predictor program is configured to generate predicted cropping characteristics for any given input image, wherein cropping characteristics for any particular input image comprise coordinates of cropping boundaries with respect to the particular input image prior to cropping, and wherein the pre-defined cropping characteristics of each respective set of training master images define one or more rectangular training bounding boxes, each enclosing a respective region of interest (ROI) of the associated training raw image;
training the ML predictor program to predict cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images;
subsequent to training the ML predictor program with the first plurality of training raw images, applying the trained ML predictor program to a sequence of video image frames in order to determine for each respective video image frame of the sequence a respective score corresponding to a highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame by the trained ML predictor program; and
selecting and storing, in non-transitory computer-readable memory, information indicative of the respective video image frame having the highest respective score.
2. The method of claim 1 , wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images comprises:
for each respective training raw image, generating respective model cropping characteristics according a computational model;
for each respective training raw image, computing respective deviations between the generated respective model cropping characteristics and the pre-defined cropping characteristics represented in the respective set of training master images associated with the respective training raw image; and
adjusting parameters of the computational model to simultaneously minimize all of the computed respective deviations.
3. The method of claim 1 , wherein the ML predictor program comprises an artificial neural network (ANN),
wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images comprises training the ANN to predict the cropping characteristics for each respective training raw image to within a threshold confidence level,
and wherein applying the trained ML predictor program to the sequence of video image frames in order to determine for each respective video image frame of the sequence the respective score corresponding to the highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame comprises:
applying the trained ANN to the sequence of video image frames to generate for each respective video image frame the predicted one or more subsets of cropping characteristics and the associated respective confidence levels; and
for each respective video image frame, selecting the predicted subset having the highest associated confidence level and setting the respective score to the associated confidence level of the selected predicted subset.
4. The method of claim 3 , wherein selecting and storing, in non-transitory computer-readable memory, the information indicative of the respective video image frame having the highest respective score comprises:
determining the highest score from among the respective scores;
identifying the respective video frame associated with the highest score; and
storing the identified respective video frame together with information identifying both the respective video frame and the subset associated with respective confidence level corresponding to the highest score.
5. The method of claim 1 , wherein inputting the first plurality of training raw images to the ML predictor program comprises:
retrieving the first plurality of training raw images from a training-image database,
wherein the training-image database comprises the first plurality of training raw images and the associated respective sets of training master images.
6. The method of claim 5 , wherein the associated respective sets of training master images comprise cropping data indicative of the respective pre-defined cropping characteristics,
and wherein the cropping data are at least one of cropped images, or coordinates defining cropping boundaries with respect to associated respective raw images in uncropped form.
7. The method of claim 1 ,
wherein each rectangular training bounding box has a respective fixed aspect ratio specified according to a set of predetermined training aspect ratios.
8. The method of claim 7 , wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images comprises training the ML predictor program to predict one or more computed rectangular bounding boxes that respectively match, to within a threshold confidence level, the one or more rectangular training bounding boxes of the respective set of training master images associated with each respective training raw image,
and wherein applying the trained ML predictor program to the sequence of video image frames in order to determine for each respective video image frame of the sequence the respective score corresponding to the highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame comprises applying the trained ML predictor program to the sequence of video frames to predict for each respective video frame one or more respective runtime rectangular bounding boxes.
9. The method of claim 8 , wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images further comprises training the ML predictor program to predict a confidence level for any given predicted bounding box,
and wherein applying the trained ML predictor program to the sequence of video image frames in order to determine for each respective video image frame of the sequence the respective score corresponding to the highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame further comprises predicting a confidence level for each of the predicted one or more respective runtime rectangular bounding boxes.
10. The method of claim 1 , further comprising providing for display on a display device a cropped version of the respective video frame having the highest respective score, wherein the cropped version of the respective video frame corresponds to the respective video frame cropped according to the associated cropping characteristics.
11. The method of claim 10 , wherein the sequence of video frames comprises digital streaming media content,
and wherein display of the cropped version of the respective video frame having the highest respective score is configured for at least one of promotional communication associated with the streaming media content, or electronic program control of the streaming media content.
12. A system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the system to carry out operations of a machine learning (ML) predictor program, wherein the operations include:
receiving as input to the ML predictor program a first plurality of training raw images, each respective training raw image of the first plurality being associated with a respective set of training master images, each training master image of a given respective set of training master images indicating respective pre-defined cropping characteristics for the associated respective training raw image, wherein the ML predictor program is configured to generate predicted cropping characteristics for any given input image, wherein cropping characteristics for any particular input image comprise coordinates of cropping boundaries with respect to the particular input image prior to cropping, and wherein the pre-defined cropping characteristics of each respective set of training master images define one or more rectangular training bounding boxes, each enclosing a respective region of interest (ROI) of the associated training raw image;
training the ML predictor program to predict cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images;
subsequent to training the ML predictor program with the first plurality of training raw images, applying the trained ML predictor program to a sequence of video image frames in order to determine for each respective video image frame of the sequence a respective score corresponding to a highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame by the trained ML predictor program; and
selecting and storing, in non-transitory computer-readable memory, information indicative of the respective video image frame having the highest respective score.
13. The system of claim 12 , wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images comprises:
for each respective training raw image, generating respective model cropping characteristics according a computational model;
for each respective training raw image, computing respective deviations between the generated respective model cropping characteristics and the pre-defined cropping characteristics represented in the respective set of training master images associated with the respective training raw image; and
adjusting parameters of the computational model to simultaneously minimize all of the computed respective deviations.
14. The system of claim 12 , wherein the ML predictor program comprises an artificial neural network (ANN),
wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images comprises training the ANN to predict the cropping characteristics for each respective training raw image to within a threshold confidence level,
and wherein applying the trained ML predictor program to the sequence of video image frames in order to determine for each respective video image frame of the sequence the respective score corresponding to the highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame comprises:
applying the trained ANN to the sequence of video image frames to generate for each respective video image frame the predicted one or more subsets of cropping characteristics and the associated respective confidence levels; and
for each respective video image frame, selecting the predicted subset having the highest associated confidence level and setting the respective score to the associated confidence level of the selected predicted subset.
15. The system of 14 , wherein selecting and storing, in non-transitory computer-readable memory, the information indicative of the respective video image frame having the highest respective score comprises:
determining the highest score from among the respective scores;
identifying the respective video frame associated with the highest score; and
storing the identified respective video frame together with information identifying both the respective video frame and the subset associated with respective confidence level corresponding to the highest score.
16. The system of claim 12 , wherein receiving as input to the ML predictor program the first plurality of training raw images comprises:
receiving the first plurality of training raw images from a training-image database,
wherein the training-image database comprises the first plurality of training raw images and the associated respective sets of training master images,
wherein the associated respective sets of training master images comprise cropping data indicative of the respective pre-defined cropping characteristics,
and wherein the cropping data are at least one of cropped images, or coordinates defining cropping boundaries with respect to associated respective raw images in uncropped form.
17. The system of claim 12 ,
wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images comprises training the ML predictor program to predict one or more computed rectangular bounding boxes that respectively match, to within a threshold confidence level, the one or more rectangular training bounding boxes of the respective set of training master images associated with each respective training raw image,
wherein applying the trained ML predictor program to the sequence of video image frames in order to determine for each respective video image frame of the sequence the respective score corresponding to the highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame comprises applying the trained ML predictor program to the sequence of video frames to predict for each respective video frame one or more respective runtime rectangular bounding boxes,
wherein training the ML predictor program to predict the cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images further comprises training the ML predictor program to predict a confidence level for any given predicted bounding box,
and wherein applying the trained ML predictor program to the sequence of video image frames in order to determine for each respective video image frame of the sequence the respective score corresponding to the highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame further comprises predicting a confidence level for each of the predicted one or more respective runtime rectangular bounding boxes.
18. The system of claim 12 , wherein the operations further include providing for display on a display device a cropped version of the respective video frame having the highest respective score, wherein the cropped version of the respective video frame corresponds to the respective video frame cropped according to the associated cropping characteristics.
19. The system of claim 18 , wherein the sequence of video frames comprises digital streaming media content,
and wherein display of the cropped version of the respective video frame having the highest respective score is configured for at least one of promotional communication associated with the streaming media content, or electronic program control of the streaming media content.
20. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors of a system, cause the system to carry out operations of a machine learning (ML) predictor program, wherein the operations include:
receiving as input to the ML predictor program a first plurality of training raw images, each respective training raw image of the first plurality being associated with a respective set of training master images, each training master image of a given respective set of training master images indicating respective pre-defined cropping characteristics for the associated respective training raw image, wherein the ML predictor program is configured to generate predicted cropping characteristics for any given input image, wherein cropping characteristics for any particular input image comprise coordinates of cropping boundaries with respect to the particular input image prior to cropping, and wherein the pre-defined cropping characteristics of each respective set of training master images define one or more rectangular training bounding boxes, each enclosing a respective region of interest (ROI) of the associated training raw image;
training the ML predictor program to predict cropping characteristics for each respective training raw image based on the pre-defined cropping characteristics represented in the associated respective set of training master images;
subsequent to training the ML predictor program with the first plurality of training raw images, applying the trained ML predictor program to a sequence of video image frames in order to determine for each respective video image frame of the sequence a respective score corresponding to a highest statistical confidence from among confidence levels respectively associated with one or more subsets of cropping characteristics predicted for the respective video image frame by the trained ML predictor program; and
selecting and storing, in non-transitory computer-readable memory, information indicative of the respective video image frame having the highest respective score.