Video and audio synchronization with dynamic frame and sample rates
A system includes a hardware processor and a memory storing a video/audio (V/A) synchronizer including video and audio encoders. The hardware processor executes the V/A synchronizer to receive raw video and audio extracted from media content, partition the raw video into video frame patches, partition the raw audio into audio samples, pre-process the video frame patches and the audio samples for encoding. The hardware processor further executes the V/A synchronizer to encode, using the video encoder, the pre-processed video frame patches to provide pre-processed and encoded video frame patches used to provide a latent representation of the raw video, encode, using the audio encoder, the pre-processed audio samples to provide pre-processed and encoded audio samples used to provide a latent representation of the raw audio, and synchronize, using the latent representations of the raw video and the raw audio, the raw audio with the raw video.
1. A system comprising:
a hardware processor; and
a memory storing a video/audio (V/A) synchronizer including a video encoder and an audio encoder;
the hardware processor configured to execute the V/A synchronizer to:
receive raw video and raw audio extracted from media content;
partition the raw video into a plurality of video frame patches;
partition the raw audio into a plurality of audio samples;
pre-process the plurality of video frame patches for encoding to provide a plurality of pre-processed video frame patches;
pre-process the plurality of audio samples for encoding to provide a plurality of pre-processed audio samples;
encode, using the video encoder, the plurality of pre-processed video frame patches to provide a plurality of pre-processed and encoded video frame patches;
encode, using the audio encoder, the plurality of pre-processed audio samples to provide a plurality of pre-processed and encoded audio samples;
provide, using one or more of the plurality of pre-processed and encoded video frame patches, a latent representation of the raw video;
provide, using the plurality of pre-processed and encoded audio samples, a latent representation of the raw audio; and
synchronize, using the latent representation of the raw video and the latent representation of the raw audio, the raw audio with the raw video.
2. The system of claim 1 , wherein all of the plurality of pre-processed and encoded video frame patches are used to provide the latent representation of the raw video.
3. The system of claim 1 , wherein at least one of the plurality of pre-processed and encoded video frame patches is not used to provide the latent representation of the raw video, and wherein the at least one of the plurality of pre-processed and encoded video frame patches is omitted from use randomly or based on attention.
4. The system of claim 1 , wherein the raw video and the raw audio are not transformed from original media specifications of the media content.
5. The system of claim 1 , wherein:
to pre-process the plurality of video frame patches, the hardware processor is further configured to execute the V/A synchronizer to project each of the plurality of video frame patches onto a respective video token to provide a plurality of tokenized video frame patches;
to pre-process the plurality of audio samples, the hardware processor is further configured to execute the V/A synchronizer to project each of the plurality of audio samples onto a respective audio token to provide a plurality of tokenized audio samples; or
a combination thereof.
6. The system of claim 5 , wherein:
to pre-process the plurality of video frame patches, the hardware processor is further configured to execute the V/A synchronizer to concatenate the plurality of tokenized video frame patches with a learnable video modality token;
to pre-process the plurality of audio samples, the hardware processor is further configured to execute the V/A synchronizer to concatenate the plurality of tokenized audio samples with a learnable audio modality token; or
a combination thereof.
7. The system of claim 5 , wherein:
to pre-process the plurality of video frame patches, the hardware processor is further configured to execute the V/A synchronizer to apply time-aware positional encoding to the plurality of tokenized video frame patches;
to pre-process the plurality of audio samples, the hardware processor is further configured to execute the V/A synchronizer to apply time-aware positional encoding to the plurality of tokenized audio samples; or
a combination thereof.
8. The system of claim 1 , wherein:
the video encoder comprises a first Transformer trained to encode video;
the audio encoder comprises a second Transformer trained to encode audio; or
a combination thereof.
9. The system of claim 1 , wherein a number of video frames included in the raw video varies based on an original frame rate of the media content, and wherein a number of audio samples included in the raw audio varies based on an original sample rate of the media content.
10. The system of claim 1 , wherein:
to synchronize the raw audio with the raw video, the hardware processor is further configured to execute the V/A synchronizer to compare the latent representation of the raw video with the latent representation of the raw audio through a contrastive loss.
11. The system of claim 1 , wherein the V/A synchronizer does not include a convolutional neural network.
12. The system of claim 1 , wherein synchronizing the raw audio with the raw video provides a first V/A synchronized media segment, and wherein the hardware processor is further configured to execute the V/A synchronizer to:
synchronize at least a second raw audio segment of the media content with at least a second raw video segment of the media content to provide at least a second V/A synchronized media segment; and
assess, based on the first V/A synchronized media segment and the at least the second V/A synchronized media segment, a V/A synchronization status of the media content as a whole.
13. A method for use by a system including a hardware processor and a memory storing a video/audio (V/A) synchronizer including a video encoder and an audio encoder, the method comprising:
receiving, by the V/A synchronizer executed by the hardware processor, raw video and raw audio extracted from media content;
partitioning, by the V/A synchronizer executed by the hardware processor, the raw video into a plurality of video frame patches;
partitioning, by the V/A synchronizer executed by the hardware processor, the raw audio into a plurality of audio samples;
pre-processing, by the V/A synchronizer executed by the hardware processor, the plurality of video frame patches for encoding to provide a plurality of pre-processed video frame patches;
pre-processing, by the V/A synchronizer executed by the hardware processor, the plurality of audio samples for encoding to provide a plurality of pre-processed audio samples;
encoding, by the V/A synchronizer executed by the hardware processor and using the video encoder, the plurality of pre-processed video frame patches to provide a plurality of pre-processed and encoded video frames;
encoding, by the V/A synchronizer executed by the hardware processor and using the audio encoder, the plurality of pre-processed audio samples to provide a plurality of pre-processed and encoded audio samples;
providing, by the V/A synchronizer executed by the hardware processor and using one or more of the plurality of pre-processed and encoded video frame patches, a latent representation of the raw video;
providing, by the V/A synchronizer executed by the hardware processor and using the plurality of pre-processed and encoded audio samples, a latent representation of the raw audio; and
synchronizing, by the V/A synchronizer executed by the hardware processor and using the latent representation of the raw video and the latent representation of the raw audio, the raw audio with the raw video.
14. The method of claim 13 , wherein all of the plurality of pre-processed and encoded video frame patches are used to provide the latent representation of the raw video.
15. The method of claim 13 , wherein at least one of the plurality of pre-processed and encoded video frame patches is not used to provide the latent representation of the raw video, and wherein the at least one of the plurality of video frame patches is omitted from use randomly or based on attention.
16. The method of claim 13 , wherein the raw video and the raw audio are not transformed from original media specifications of the media content.
17. The method of claim 13 , further comprising:
pre-processing the plurality of video frame patches includes projecting, by the V/A synchronizer executed by the hardware processor, each of the plurality of video frame patches onto a respective video token to provide a plurality of tokenized video frame patches;
pre-processing the plurality of audio samples by projecting, by the V/A synchronizer executed by the hardware processor, each of the plurality of audio samples onto a respective audio token to provide a plurality of tokenized audio samples; or
a combination thereof.
18. The method of claim 17 , further comprising:
pre-processing the plurality of video frame patches by concatenating, by the V/A synchronizer executed by the hardware processor, the plurality of tokenized video frame patches with a learnable video modality token; and
pre-processing the plurality of audio samples by concatenating, by the V/A synchronizer executed by the hardware processor, the plurality of tokenized audio samples with a learnable audio modality token; or
a combination thereof.
19. The method of claim 17 , further comprising:
pre-processing the plurality of video frame patches by applying, by the V/A synchronizer executed by the hardware processor, time-aware positional encoding to the plurality of tokenized video frame patches; and
pre-processing the plurality of audio samples by applying, by the V/A synchronizer executed by the hardware processor, time-aware positional encoding to the plurality of tokenized audio samples; or
a combination thereof.
20. The method of claim 13 , wherein:
the video encoder comprises a first Transformer trained to encode video;
the audio encoder comprises a second Transformer trained to encode audio; or
a combination thereof.
21. The method of claim 13 , wherein how many video frames are included in the raw video varies based on an original frame rate of the media content, and wherein how many audio samples are included in the raw audio varies based on an original sample rate of the media content.
22. The method of claim 13 , wherein synchronizing the raw audio with the raw video comprises comparing the latent representation of the raw video with the latent representation of raw audio through a contrastive loss.
23. The method of claim 13 , wherein the V/A synchronizer does not include a convolutional neural network.
24. The method of claim 13 , wherein synchronizing the raw audio with the raw video provides a first V/A synchronized media segment, the method further comprising:
synchronizing, by the V/A synchronizer executed by the hardware processor, at least a second raw audio segment of the media content with a respective at least a second raw video segment of the media content to provide at least a second V/A synchronized media segment; and
assessing, by the V/A synchronizer executed by the hardware processor based on the first V/A synchronized media segment and the at least the second V/A synchronized media segment, a V/A synchronization status of the media content as a whole.