Ensemble of machine learning models for automatic scene change detection
Techniques for automatic scene change detection are described. As one example, a computer-implemented method includes receiving a request to train an ensemble of machine learning models on a training dataset of videos having labels that indicate scene changes to detect a scene change in a video, partitioning each video file of the training dataset of videos into a plurality of shots, training the ensemble of machine learning models into a trained ensemble of machine learning models based at least in part on the plurality of shots of the training dataset of videos and the labels that indicate scene changes, receiving an inference request for an input video, partitioning the input video into a plurality of shots, generating, by the trained ensemble of machine learning models, an inference of one or more scene changes in the input video based at least in part on the plurality of shots of the input video, and transmitting the inference to a client application or to a storage location.
1 . A computer-implemented method comprising:
training a machine learning model, on a training dataset of scene change data, to predict when a shot of a video is an end of a scene in the video;
generating, by the trained machine learning model based on an input of shot representations of a plurality of shots of an input video, a prediction that a shot of the plurality of shots is an end of a scene in the input video;
inserting secondary content into the input video at a shot boundary of the shot indicated by the prediction as the end of the scene to generate an output video; and
transmitting the output video to an application or to a storage location.
2 . The computer-implemented method of claim 1 , wherein the shot representations of the plurality of shots of the input video comprise audio features and video features.
3 . The computer-implemented method of claim 2 , further comprising concatenating the audio features and the video features into a multimodal shot representation, wherein the input of the trained machine learning model comprises the multimodal shot representation.
4 . The computer-implemented method of claim 2 , further comprising extracting voice features from corresponding audio of the input video.
5 . The computer-implemented method of claim 2 , wherein the trained machine learning model comprises a video machine learning model that inputs the audio features of the input video, and an audio machine learning model that inputs the video features of the input video.
6 . The computer-implemented method of claim 1 , wherein the training dataset comprises a timestamp for a training video.
7 . The computer-implemented method of claim 1 , wherein the shot predicted to be the end of the scene is the end of a first scene that is followed by a second different scene in the input video.
8 . The computer-implemented method of claim 1 , wherein the scene in the input video is a set of shots filming action in a same time and location.
9 . The computer-implemented method of claim 1 , further comprising:
receiving a request for the output video from a client device; and
sending the output video to the client device.
10 . A non-transitory computer-readable medium storing code that, when executed by a device, causes the device to perform a method comprising:
training a machine learning model, on a training dataset of scene change data, to predict when a shot of a video is an end of a scene in the video;
generating, by the trained machine learning model based on an input of shot representations of a plurality of shots of an input video, a prediction that a shot of the plurality of shots is an end of a scene in the input video;
inserting secondary content into the input video at a shot boundary of the shot indicated by the prediction as the end of the scene to generate an output video; and
transmitting the output video to an application or to a storage location.
11 . The non-transitory computer-readable medium of claim 10 , wherein the shot representations of the plurality of shots of the input video comprise audio features and video features.
12 . The non-transitory computer-readable medium of claim 11 , wherein the method further comprises concatenating the audio features and the video features into a multimodal shot representation, wherein the input of the trained machine learning model comprises the multimodal shot representation.
13 . The non-transitory computer-readable medium of claim 11 , wherein the method further comprises extracting voice features from corresponding audio of the input video.
14 . The non-transitory computer-readable medium of claim 11 , wherein the trained machine learning model comprises a video machine learning model that inputs the audio features of the input video, and an audio machine learning model that inputs the video features of the input video.
15 . The non-transitory computer-readable medium of claim 10 , wherein the training dataset comprises a timestamp for a training video.
16 . The non-transitory computer-readable medium of claim 10 , wherein the shot predicted to be the end of the scene is the end of a first scene that is followed by a second different scene in the input video.
17 . The non-transitory computer-readable medium of claim 10 , wherein the scene in the input video is a set of shots filming action in a same time and location.
18 . The non-transitory computer-readable medium of claim 10 , wherein the method further comprises:
receiving a request for the output video from a client device; and
sending the output video to the client device.