Alignment of alternate dialogue audio track to frames in a multimedia production using background audio matching
A computer-implemented audio alignment system includes a storage device and a processor. The storage device is configured to ingest and store video files, which each include one or more respective audio soundtracks. The processor is configured to receive an alternate audio file associated with one of the video files. The alternate audio file includes an alternate audio soundtrack with dialogue and non-dialogue audio. The video file includes an original language audio soundtrack with dialogue and non-dialogue audio. The processor aligns the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack and aligns the alternate audio soundtrack with the film based on the alignment of the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
1. A computer-implemented audio alignment system comprising
a data storage device configured to ingest and store one or more video files thereon, wherein the one or more video files comprise one or more respective audio soundtracks; and
one or more processors configured with instructions to
receive an alternate audio file associated with a video file of the one or more video files, wherein the alternate audio file comprises an alternate audio soundtrack with dialogue audio and non-dialogue audio and the video file comprises a film and an original language audio soundtrack, wherein the original language audio soundtrack comprises dialogue audio and non-dialogue audio;
filter to remove the dialogue audio from the alternate audio soundtrack and from the original language soundtrack;
align the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack based on a filtered alternate audio soundtrack and a filtered original language soundtrack; and
align the alternate audio soundtrack with the film based on the alignment of the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
2. The computer-implemented audio alignment system of claim 1 , wherein the one or more processors are further configured with instructions to filter the dialogue audio from the alternate audio soundtrack and from the original language soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language soundtrack.
3. The computer-implemented audio alignment system of claim 1 , wherein the one or more processors are further configured with instructions to
identify sound signatures in the non-dialogue audio from the alternate audio soundtrack and the non-dialogue audio from the original language audio soundtrack;
determine matching sound signatures from the non-dialogue audio from the alternate audio soundtrack and the non-dialogue audio from the original language audio soundtrack;
assess alignment between matching sound signatures to determine if there is an offset between the non-dialogue audio from the alternate audio soundtrack and the non-dialogue audio from the original language audio soundtrack; and
align the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack when there is an offset.
4. The computer-implemented audio alignment system of claim 3 , wherein the alignment assessment comprises determining an alignment confidence score.
5. The computer-implemented audio alignment system of claim 3 , wherein the one or more processors are further configured with instructions to
identify the sound signatures as patterns of spikes in amplitude; or
identify the sound signatures as unique features of audio signals in each of the alternate audio soundtrack and the original language audio soundtrack that is distinct from other features of each audio signal.
6. The computer-implemented audio alignment system of claim 1 , wherein the non-dialogue audio from both the alternate audio soundtrack and the original language audio soundtrack comprises sound effects and music.
7. The computer-implemented audio alignment system of claim 1 , wherein the alternate audio soundtrack and the original language soundtrack each comprise two or more separate audio tracks.
8. The computer-implemented audio alignment system of claim 7 , wherein the one or more processors are further configured with instructions to remove the dialogue audio track for the alternate audio soundtrack and the dialogue audio track for the original language audio soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
9. The computer-implemented audio alignment system of claim 7 , wherein the aligning of the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack further comprises aligning a sound effects audio track, a music audio track, or both, from the alternate audio soundtrack with a respective sound effects audio track, a respective music audio track, or both, from the original language audio soundtrack.
10. The computer-implemented audio alignment system of claim 1 , wherein the alternate audio soundtrack and the original language audio soundtrack each comprise a single audio track for dialogue, sound effects, and music.
11. The computer-implemented audio alignment system of claim 10 , wherein the one or more processors are further configured with instructions to
identify the dialogue audio in both the alternate audio soundtrack and the original language audio soundtrack; and
mask the dialogue audio for both the alternate audio soundtrack and the original language audio soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
12. The computer-implemented audio alignment system of claim 11 , wherein identifying dialogue audio in both the alternate audio soundtrack and the original language audio soundtrack comprises identifying audio signals in the respective audio soundtracks that are within a mid-range frequency.
13. A method implemented on a computer system for aligning audio soundtracks, wherein one or more processors in the computer system is particularly configured to perform a number of processing steps comprising
receiving an original language film with an original language audio soundtrack, wherein the original language soundtrack comprises dialogue audio and background audio;
receiving an alternate audio soundtrack associated with the original language film, wherein the alternate audio soundtrack comprises dialogue audio and background audio;
filtering to remove the dialogue audio from the alternate audio soundtrack and from the original language soundtrack;
aligning the background audio from the alternate audio soundtrack with the background audio from the original language audio soundtrack based on a filtered alternate audio soundtrack and a filtered original language soundtrack; and
aligning the alternate audio soundtrack with the original language film based on the alignment of the background audio from the alternate audio soundtrack with the background audio from the original language audio soundtrack.
14. The method of claim 13 , wherein the one or more processors are configured to perform the processing step of filtering the dialogue audio from the alternate audio soundtrack and from the original language soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language soundtrack.
15. The method of claim 13 , wherein the one or more processors is further configured to perform additional processing steps comprising
identifying sound signatures in the background audio from the alternate audio soundtrack and in the background audio from the original language audio soundtrack;
determining matching sound signatures from the background audio from the alternate audio soundtrack and from the background audio from the original language audio soundtrack;
assessing alignment between matching sound signatures to determine whether there is an offset between the background audio from the alternate audio soundtrack and the background audio from the original language audio soundtrack; and
aligning the background audio from the alternate audio soundtrack with the background audio from the original language audio soundtrack when there is an offset.
16. The method of claim 15 , wherein the step of assessing alignment further comprises calculating a confidence score to determine whether there is an offset.
17. The method of claim 15 , wherein the one or more processors are further configured to perform additional processing steps comprising
identifying the sound signatures as patterns of spikes in amplitude; or
identifying the sound signatures as unique features of audio signals in each of the alternate audio soundtrack and the original language audio soundtrack that is distinct from other features of each audio signal.
18. The method of claim 13 , wherein the non-dialogue audio from both the alternate audio soundtrack and the original language audio soundtrack comprises sound effects and music.
19. The method of claim 13 , wherein the alternate audio soundtrack and the original language soundtrack comprise two or more separate audio tracks.
20. The method of claim 19 , wherein the one or more processors are further configured to perform an additional processing step comprising removing the dialogue audio track for the alternate audio soundtrack and the dialogue audio track for the original language audio soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
21. The method of claim 19 , wherein the step of aligning of the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack further comprises aligning a sound effects audio track, a music audio track, or both, from the alternate audio soundtrack with a respective sound effects audio track, a respective music audio track, or both, from the original language audio soundtrack.
22. The method of claim 13 , wherein the alternate audio soundtrack and the original language audio soundtrack each comprise a single audio track for dialogue, sound effects, and music.
23. The method of claim 22 , wherein the one or more processors are further configured to perform additional processing steps comprising
identifying the dialogue audio in both the alternate audio soundtrack and the original language audio soundtrack; and
masking the dialogue audio for both the alternate audio soundtrack and the original language audio soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
24. The method of claim 22 , wherein the step of identifying the dialogue audio in both the alternate audio soundtrack and the original language audio soundtrack further comprises identifying audio signals in the respective audio soundtracks that are within a mid-range frequency.
25. A non-transitory computer readable storage medium containing instructions for instantiating a special purpose computer to align audio soundtracks, wherein the instructions implement a computer process comprising the steps of
receiving an original language film with an original language audio soundtrack, wherein the original language audio soundtrack comprises dialogue audio and background audio;
receiving an alternate soundtrack associated with the original language film, wherein the alternate soundtrack comprises dialogue audio and background audio;
filtering to remove the dialogue audio from the alternate audio soundtrack and from the original language soundtrack;
determining matching sound signatures from a filtered original language audio soundtrack and a filtered alternate soundtrack generated in the filtering step; and
aligning the background audio from the alternate audio soundtrack with the background audio from the original language soundtrack.
26. The non-transitory computer readable storage medium of claim 25 , wherein the computer process further comprises filtering the dialogue audio from the alternate audio soundtrack and from the original language soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language soundtrack.
27. The non-transitory computer readable storage medium of claim 25 , wherein the computer process further comprises
identifying sound signatures in the background audio from the alternate audio soundtrack and in the background audio from the original language soundtrack;
determining matching sound signatures from the background audio from the alternate audio soundtrack and from the background audio from the original language soundtrack;
assessing alignment between matching sound signatures to determine whether there is an offset between the background audio from the alternate audio soundtrack and the background audio from the original language soundtrack; and
aligning the background audio from the alternate audio soundtrack with the background audio from the original language soundtrack when there is an offset.
28. The non-transitory computer readable storage medium of claim 25 , wherein the step of assessing alignment further comprises calculating a confidence score to determine whether there is an offset.
29. The non-transitory computer readable storage medium of claim 25 , wherein the computer process further comprises
identifying the sound signatures as patterns of spikes in amplitude; or
identifying the sound signatures as unique features of audio signals in each of the alternate audio soundtrack and the original language audio soundtrack that is distinct from other features of each audio signal.
30. The non-transitory computer readable storage medium of claim 25 , wherein the non-dialogue audio from both the alternate audio soundtrack and the original language audio soundtrack comprises sound effects and music.
31. The non-transitory computer readable storage medium of claim 25 , wherein the alternate audio soundtrack and the original language soundtrack comprise two or more separate audio tracks.
32. The non-transitory computer readable storage medium of claim 31 , wherein the computer process further comprises removing the dialogue audio track for the alternate audio soundtrack and the dialogue audio track for the original language audio soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
33. The non-transitory computer readable storage medium of claim 31 , wherein the step of aligning of the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack further comprises aligning a sound effects audio track, a music audio track, or both, from the alternate audio soundtrack with a respective sound effects audio track, a respective music audio track, or both, from the original language audio soundtrack.
34. The non-transitory computer readable storage medium of claim 25 , wherein the alternate audio soundtrack and the original language audio soundtrack each comprise a single audio track for dialogue, sound effects, and music.
35. The non-transitory computer readable storage medium of claim 34 , wherein the computer process further comprises
identifying the dialogue audio in both the alternate audio soundtrack and the original language audio soundtrack; and
masking the dialogue audio for both the alternate audio soundtrack and the original language audio soundtrack before aligning the non-dialogue audio from the alternate audio soundtrack with the non-dialogue audio from the original language audio soundtrack.
36. The non-transitory computer readable storage medium of claim 35 , wherein the step of identifying the dialogue audio in both the alternate audio soundtrack and the original language audio soundtrack further comprises identifying audio signals in the respective audio soundtracks that are within a mid-range frequency.