Techniques for identifying synchronization errors in media titles
A neural network system that is trained to identify one or more portions of a media title where synchronization errors are likely to be present. The neural network system is trained based on a first set of media titles where synchronization errors are present and a second set of media titles where synchronization errors are absent. The second set of media titles can be generated by introducing synchronization errors into a set of media titles that otherwise lack synchronization errors. Via training, the neural network system learns to identify specific visual features included in one or more video frames and corresponding audio features that should be played back in synchrony with the associated visual features. Accordingly, when presented with a media title that includes synchronization errors, the neural network can indicate the specific frames where synchronization errors are likely to be present.
1. A neural network system implemented by one or more computers,
wherein the neural network system identifies one or more blocks in a media clip that include audio data that is misaligned with corresponding video data,
wherein the neural network system comprises:
a convolutional subnetwork that generates a plurality of feature maps by generating, for each block included in a plurality of blocks in the media clip, a corresponding feature map derived from both one or more audio features and also one or more video features of the block in the media clip;
an attention module that:
computes a first set of data based on the plurality of feature maps;
executes one or more convolution operations to compute a plurality of confidence values corresponding to the plurality of blocks in the media clip based on the first set of data;
generates a plurality of weight values corresponding to the plurality of blocks based on the plurality of confidence values; and
computes a weighted average based on the plurality of weight values to generate a global feature vector; and
an output layer that identifies, based on the global feature vector from the attention module, a first block included in the plurality of blocks that includes first audio data that is misaligned with corresponding first video data.
2. The neural network system of claim 1 , wherein the convolutional subnetwork comprises:
a plurality of three-dimensional convolutional networks that share a first set of weights, wherein each three-dimensional convolutional network included in the plurality of three-dimensional convolutional networks includes one or more audio feature extraction layers, one or more video feature extraction layers, and one or more joint audio/video extraction layers.
3. The neural network system of claim 1 , wherein the attention module comprises:
a convolution layer that performs the one or more convolution operations on the first set of data to generate the plurality of confidence values;
a softmax layer that generates the plurality of weight values based on the plurality of confidence values, wherein the softmax layer performs a normalization operation on the plurality of confidence values to generate the plurality of weight values; and
a weighted averaging layer that generates the global feature vector for the media clip based on the plurality of weight values, wherein the global feature vector indicates that the first block in the plurality of blocks includes the first audio data that is misaligned with the corresponding first video data.
4. The neural network system of claim 3 , wherein the attention module further comprises:
a global pooling layer that generates a different feature vector for each feature map included in the plurality of feature maps to produce the first set of data, wherein the convolution layer computes each confidence value included in the plurality of confidence values based on a corresponding feature vector, and wherein the global feature vector indicates a first region within the first block in the plurality of blocks where the first audio data is misaligned with the corresponding first video data.
5. The neural network system of claim 1 , wherein the output layer includes one or more fully connected layers that perform a binary classification based on the plurality of weight values to generate a first classification for the first block, wherein the first classification indicates that the first audio data is misaligned with the corresponding first video data.
6. The neural network system of claim 1 , wherein a first feature map corresponding to the first block of the plurality of blocks includes a first joint audio/visual feature that is derived from a first audio feature and a first video feature.
7. The neural network system of claim 6 , wherein the first audio feature corresponds to a first sound that is played during playback of the media clip, the first video feature corresponds to a first event that is depicted during playback of the media clip, and the first sound is played back in conjunction with the first event in the absence of misalignment between audio data and corresponding video data.
8. The neural network system of claim 6 , wherein the first audio feature corresponds to a first sound that is played during playback of the media clip, the first video feature corresponds to a first event that is depicted during playback of the media clip, and the first sound is not played back in conjunction with the first event in the presence of misalignment between audio data and corresponding video data.
9. The neural network system of claim 6 , wherein the first video feature corresponds to a first intersection between a first set of pixels and a second set of pixels, and the first audio feature corresponds to a sound associated with the first intersection.
10. The neural network system of claim 1 , wherein at least one of the convolutional subnetwork, the attention module, or the output layer is trained based on a first set of media clips that do not include misalignment between audio data and corresponding video data and a second set of media clips that include misalignment between audio data and corresponding video data.
11. The neural network system of claim 1 , wherein the attention module comprises a softmax layer that performs a normalization operation on the plurality of confidence values to generate the plurality of weight values.
12. A computer-implemented method, comprising:
identifying, via a neural network, one or more blocks in a media clip that include audio data that is misaligned with corresponding video data, wherein the neural network is configured to:
generate a plurality of feature maps by generating, for each block included in a plurality of blocks in the media clip, a corresponding feature map derived from both one or more audio features and also one or more video features of the block in the media clip,
compute a first set of data based on the plurality of feature maps,
execute one or more convolution operations to compute a plurality of confidence values corresponding to the plurality of blocks in the media clip,
generate a plurality of weight values corresponding to the plurality of blocks based on the plurality of confidence values,
compute a weighted average based on the plurality of weight values to generate a global feature vector, and
identify, based on the global feature vector, a first block included in the plurality of blocks that includes first audio data that is misaligned with corresponding first video data.
13. The computer-implemented method of claim 12 , further comprising:
performing the one or more convolution operations on the first set of data to generate the plurality of confidence values;
generating the plurality of weight values based on the plurality of confidence values computed, wherein a softmax layer performs a normalization operation on the plurality of confidence values to generate the plurality of weight values; and
generating the global feature vector for the media clip based on the plurality of weight values generated, wherein the global feature vector indicates that the first block in the plurality of blocks includes the first audio data that is misaligned with the corresponding first video data.
14. The computer-implemented method of claim 13 , further comprising generating, via global average pooling, a different feature vector for each feature map included in the plurality of feature maps to produce the first set of data, wherein computing each confidence value included in the plurality of confidence values is performed based on a corresponding feature vector, and wherein the global feature vector indicates a first region within the first block in the plurality of blocks where the first audio data is misaligned with the corresponding first video data.
15. The computer-implemented method of claim 12 , wherein a first feature map corresponding to the first block of the plurality of blocks includes a first joint audio/visual feature that is derived from a first audio feature and a first video feature.
16. The computer-implemented method of claim 15 , wherein the first audio feature corresponds to a first sound that is played during playback of the media clip, the first video feature corresponds to a first event that is depicted during playback of the media clip, and the first sound is played back in conjunction with the first event in the absence of misalignment between audio data and corresponding video data.
17. A non-transitory computer-readable medium storing program instructions that, when executed by a processor, cause the processor to perform the steps of:
generating a plurality of feature maps by generating, for each block included in a plurality of blocks in a media clip, a corresponding feature map derived from both one or more audio features and also one or more video features of the block in the media clip;
computing a first set of data based on the plurality of feature maps;
executing one or more convolution operations to compute a plurality of confidence values corresponding to the plurality of blocks in the media clip;
generating a plurality of weight values corresponding to the plurality of blocks based on the plurality of confidence values;
computing a weighted average based on the plurality of weight values to generate a global feature vector; and
identifying, based on the global feature vector, a first block included in the plurality of blocks that includes first audio data that is misaligned with corresponding first video data.
18. The non-transitory computer-readable medium of claim 17 , wherein a first audio feature corresponds to a first sound that is played during playback of the media clip, a first video feature corresponds to a first event that is depicted during playback of the media clip, and the first sound is played back in conjunction with the first event in the absence of misalignment between audio data and corresponding video data.
19. The non-transitory computer-readable medium of claim 18 , wherein the first audio feature corresponds to a first sound that is played during playback of the media clip, the first video feature corresponds to a first event that is depicted during playback of the media clip, and the first sound is not played back in conjunction with the first event in the presence of misalignment between audio data and corresponding video data.
20. The non-transitory computer-readable medium of claim 18 , wherein the first video feature corresponds to a first intersection between a first set of pixels and a second set of pixels, and the first audio feature corresponds to a sound associated with the first intersection.
21. The non-transitory computer-readable medium of claim 17 , wherein the program instructions further comprise program instructions that, when executed by the processor, cause the processor to perform a step of training a neural network to identify the first block based on a first set of media clips that do not include misalignment between audio data and corresponding video data and a second set of media clips that include misalignment between audio data and corresponding video data.