Gating model for video analysis
Implementations described herein relate to methods, devices, and computer-readable media to perform gating for video analysis. In some implementations, a computer-implemented method includes obtaining a video comprising a plurality of frames and corresponding audio. The method further includes performing sampling to select a subset of the plurality of frames based on a target frame rate and extracting a respective audio spectrogram for each frame in the subset of the plurality of frames. The method further includes reducing resolution of the subset of the plurality of frames. The method further includes applying a machine-learning based gating model to the subset of the plurality of frames and corresponding audio spectrograms and obtaining, as output of the gating model, an indication of whether to analyze the video to add one or more video annotations.
1. A computer-implemented method to train a machine-learning based gating model to generate an indication of whether to analyze a video to add annotations corresponding to a particular feature, wherein the machine-learning based gating model comprises:
a first model that comprises a first convolutional neural network that generates a likelihood that the particular feature is present in a video based on video frames of the video; and
a second model that receives as input the likelihood that the particular feature is present in the video and generates the indication, the method comprising:
obtaining a training set comprising:
a plurality of training videos, wherein each training video comprises a plurality of frames, and wherein each training video is a low-resolution, sampled version of a corresponding high-resolution video; and
a plurality of training labels, each training label indicative of presence of the particular feature in the high-resolution videos corresponding to the one or more of the plurality of training videos; and
training the gating model, wherein the training includes, for each training video in the training set,
generating, by application of the first model to the training video, a likelihood that the particular feature is present in the training video;
generating based on the likelihood that the particular feature is present in the training video, by application of the second model, the indication of whether to analyze the training video to add annotations corresponding to a particular feature;
generating feedback data based on the training labels associated with the corresponding high-resolution video and the indication; and
providing the feedback data as a training input to the first model and to the second model.
2. The computer-implemented method of claim 1 , wherein the particular feature includes at least one of a human face, a type of movement, or a type of object.
3. The computer-implemented method of claim 1 , wherein the plurality of training videos in the training set include at least one video in which the particular feature is present and at least one video in which the particular feature is absent, and wherein training the gating model comprises one or more of automatically adjusting a weight of one or more nodes of the first convolutional neural network of the first model or automatically adjusting a connectivity between one or more pairs of nodes of the first convolutional neural network of the first model.
4. The computer-implemented method of claim 1 , wherein the second model includes one or more of a heuristics-based model, a recurrent neural network, or a Markov chain analysis model, and wherein training the gating model comprises one or more of automatically adjusting one or more parameters of the heuristics-based model, the recurrent neural network, or the Markov chain analysis model.
5. The computer-implemented method of claim 1 , wherein training the gating model further comprises dividing the plurality of frames of the training video into a plurality of stacks of frames, each stack comprising at least one frame, wherein the plurality of stacks are organized in an ordered sequence, and wherein training the gating model is performed sequentially for each stack of frames in the plurality of stacks of frames.
6. The computer-implemented method of claim 5 , wherein the second model is configured to store the generated indication for each stack of the training video, and wherein generating the indication for a particular stack is further based on respective stored indications for one or more prior stacks in the ordered sequence.
7. The computer-implemented method of claim 1 , wherein one or more training videos of the plurality of training videos further comprise audio spectrograms corresponding to the plurality of frames, and wherein the first model further comprises:
a second convolutional neural network that is trained to analyze audio spectrograms; and
a fusion network that receives output of the first convolutional neural network and the second convolutional neural network as inputs, and generates the likelihood that the particular feature is present in the video.
8. A computing device comprising:
a processor; and
a memory, with instructions stored thereon that, when executed by the processor cause the processor to perform operations to train a machine-learning based gating model to generate an indication of whether to analyze a video to add annotations corresponding to a particular feature, wherein the machine-learning based gating model comprises:
a first model that comprises a first convolutional neural network that generates a likelihood that the particular feature is present in a video based on video frames of the video; and
a second model that receives as input the likelihood that the particular feature is present in the video and generates the indication,
and wherein the operations comprise:
obtaining a training set comprising:
a plurality of training videos, wherein each training video comprises a plurality of frames, and wherein each training video is a low-resolution, sampled version of a corresponding high-resolution video; and
a plurality of training labels, each training label indicative of presence of the particular feature in the high-resolution videos corresponding to the one or more of the plurality of training videos; and
training the gating model, wherein the training includes, for each training video in the training set,
generating, by application of the first model to the training video, a likelihood that the particular feature is present in the training video;
generating based on the likelihood that the particular feature is present in the training video, by application of the second model, the indication of whether to analyze the training video to add annotations corresponding to a particular feature;
generating feedback data based on the training labels associated with the corresponding high-resolution video and the indication; and
providing the feedback data as a training input to the first model and to the second model.
9. The computing device of claim 8 , wherein the particular feature includes at least one of a human face, a type of movement, or a type of object.
10. The computing device of claim 8 , wherein the plurality of training videos in the training set include at least one video in which the particular feature is present and at least one video in which the particular feature is absent, and wherein training the gating model comprises one or more of automatically adjusting a weight of one or more nodes of the first convolutional neural network of the first model or automatically adjusting a connectivity between one or more pairs of nodes of the first convolutional neural network of the first model.
11. The computing device of claim 8 , wherein the second model includes one or more of a heuristics-based model, a recurrent neural network, or a Markov chain analysis model, and wherein training the gating model comprises one or more of automatically adjusting one or more parameters of the heuristics-based model, the recurrent neural network, or the Markov chain analysis model.
12. The computing device of claim 8 , wherein training the gating model further comprises dividing the plurality of frames of the training video into a plurality of stacks of frames, each stack comprising at least one frame, wherein the plurality of stacks are organized in an ordered sequence, and wherein training the gating model is performed sequentially for each stack of frames in the plurality of stacks of frames.
13. The computing device of claim 12 , wherein the second model is configured to store the generated indication for each stack of the training video, and wherein generating the indication for a particular stack is further based on respective stored indications for one or more prior stacks in the ordered sequence.
14. The computing device of claim 8 , wherein one or more training videos of the plurality of training videos further comprise audio spectrograms corresponding to the plurality of frames, and wherein the first model further comprises:
a second convolutional neural network that is trained to analyze audio spectrograms; and
a fusion network that receives output of the first convolutional neural network and the second convolutional neural network as inputs, and generates the likelihood that the particular feature is present in the video.
15. A non-transitory computer-readable medium with instructions stored thereon that, when executed by a computer, cause the computer to perform operations to train a machine-learning based gating model to generate an indication of whether to analyze a video to add annotations corresponding to a particular feature, wherein the machine-learning based gating model comprises:
a first model that comprises a first convolutional neural network that generates a likelihood that the particular feature is present in a video based on video frames of the video; and
a second model that receives as input the likelihood that the particular feature is present in the video and generates the indication,
and wherein the operations comprise:
obtaining a training set comprising:
a plurality of training videos, wherein each training video comprises a plurality of frames, and wherein each training video is a low-resolution, sampled version of a corresponding high-resolution video; and
a plurality of training labels, each training label indicative of presence of the particular feature in the high-resolution videos corresponding to the one or more of the plurality of training videos; and
training the gating model, wherein the training includes, for each training video in the training set,
generating, by application of the first model to the training video, a likelihood that the particular feature is present in the training video;
generating based on the likelihood that the particular feature is present in the training video, by application of the second model, the indication of whether to analyze the training video to add annotations corresponding to a particular feature;
generating feedback data based on the training labels associated with the corresponding high-resolution video and the indication; and
providing the feedback data as a training input to the first model and to the second model.
16. The non-transitory computer-readable medium of claim 15 , wherein the particular feature includes at least one of a human face, a type of movement, or a type of object.
17. The non-transitory computer-readable medium of claim 15 , wherein the plurality of training videos in the training set include at least one video in which the particular feature is present and at least one video in which the particular feature is absent, and wherein training the gating model comprises one or more of automatically adjusting a weight of one or more nodes of the first convolutional neural network of the first model or automatically adjusting a connectivity between one or more pairs of nodes of the first convolutional neural network of the first model.
18. The non-transitory computer-readable medium of claim 15 , wherein the second model includes one or more of a heuristics-based model, a recurrent neural network, or a Markov chain analysis model, and wherein training the gating model comprises one or more of automatically adjusting one or more parameters of the heuristics-based model, the recurrent neural network, or the Markov chain analysis model.
19. The non-transitory computer-readable medium of claim 15 , wherein training the gating model further comprises dividing the plurality of frames of the training video into a plurality of stacks of frames, each stack comprising at least one frame, wherein the plurality of stacks are organized in an ordered sequence, and wherein training the gating model is performed sequentially for each stack of frames in the plurality of stacks of frames.
20. The non-transitory computer-readable medium of claim 15 , wherein one or more training videos of the plurality of training videos further comprise audio spectrograms corresponding to the plurality of frames, and wherein the first model further comprises:
a second convolutional neural network that is trained to analyze audio spectrograms; and
a fusion network that receives output of the first convolutional neural network and the second convolutional neural network as inputs, and generates the likelihood that the particular feature is present in the video.