Systems and methods for improved lip dubbing
A computer implemented system is described in various embodiments herein, the system includes a processor, a memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to: receive an initial video data object V comprising a plurality of image frame data objects I; receive a set of time-synchronized viseme parameters corresponding to target audio data object A′; and process the initial video data object V and the set of time-synchronized viseme parameters using a machine learning network to generate an output video data object V′, wherein initial mouth regions in the initial video data object V have been replaced with replacement mouth regions generated based on the set of time-synchronized viseme parameters.
1 . A computer-implemented system for generating an output video for lip dubbing, the system comprising:
a processor;
a memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to:
receive an initial video data object V comprising a plurality of image frame data objects I;
extract, from an initial audio data object A corresponding to initial video data object V, an initial set of expression/viseme codes corresponding to spectrogram segments in the initial audio data object A;
receive a set of time-synchronized viseme parameters corresponding to target audio data object A′; and
process the initial video data object V and the set of time-synchronized viseme parameters using a machine learning network to generate an output video data object V′, wherein initial mouth regions in the initial video data object V have been replaced with replacement mouth regions generated based on the set of time-synchronized viseme parameters;
wherein the initial video data object V is utilized to populate a viseme library of available visemes based on the initial set of expression/viseme codes; and
wherein the processor is configured to compare visemes identified in the target audio data object A′ to the viseme library of available visemes based on the initial set of expression/viseme codes to identify a subset of visemes not in the viseme library and requiring synthesis.
2 . The system of claim 1 , wherein the set of time-synchronized viseme parameters corresponding to the target audio data object A′ are a set of parameters identified using a plurality of encoder networks adapted to disentangle expression e and pose p from a plurality of image properties.
3 . The system of claim 2 , wherein the plurality of encoder networks are iteratively trained to disentangle the expression parameters e, the pose parameters p and residual parameters r from the plurality of image frame data objects I.
4 . The system of claim 1 , wherein the subset of visemes not in the viseme library are synthesized from visemes obtained from other work from a same actor.
5 . The system of claim 1 , wherein the subset of visemes not in the viseme library are synthesized from an eigenface.
6 . The system of claim 5 , wherein the processor is configured to generate new expression/viseme codes corresponding to the subset of visemes not in the viseme library.
7 . The system of claim 1 , wherein the processor is configured to generate the output video data object V′ by replacing at least one from the plurality of image frame data objects I in its entirety with a replacement image frame data object I′.
8 . The system of claim 1 , wherein the processor is configured to generate the output video data object V′ by replacing a portion of at least one from the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I′, wherein the portion corresponds to a mouth region.
9 . The system of claim 1 , wherein the processor is configured to generate the output video data object V′ iteratively for each character present in the initial video data object V.
10 . The system of claim 1 , wherein the machine learning network comprise a generative adversarial network.
11 . A computer-generated method for generating output video, the method comprising:
receiving an initial video data object V comprising a plurality of image frame data objects I;
extracting, from an initial audio data object A corresponding to initial video data object V, an initial set of expression/viseme codes corresponding to spectrogram segments in the initial audio data object A;
receiving a set of time-synchronized viseme parameters corresponding to target audio data object A′; and
processing the initial video data object V and the set of time-synchronized viseme parameters using a machine learning network to generate an output video data object V′, wherein initial mouth regions in the initial video data object V have been replaced with replacement mouth regions generated based on the set of time-synchronized viseme parameters;
wherein the initial video data object V is utilized to populate a viseme library of available visemes based on the initial set of expression/viseme codes; and
wherein the method further comprises comparing visemes identified in the target audio data object A′ to the viseme library of available visemes based on the initial set of expression/viseme codes to identify a subset of visemes not in the viseme library and requiring synthesis.
12 . The method of claim 11 , wherein the set of time-synchronized viseme parameters corresponding to the target audio data object A′ are a set of parameters identified using a plurality of encoder networks adapted to disentangle expression e and pose p from a plurality of image properties.
13 . The method of claim 12 , wherein the plurality of encoder networks are iteratively trained to disentangle the expression parameters e, the pose parameters p and residual parameters r from the plurality of image frame data objects I.
14 . The method of claim 11 , wherein the subset of visemes not in the viseme library are synthesized from visemes obtained from other work from a same actor.
15 . The method of claim 11 , wherein the subset of visemes not in the viseme library are synthesized from an eigenface.
16 . The method of claim 11 , further comprising generating the output video data object V′ by replacing at least one from the plurality of image frame data objects I in its entirety with a replacement image frame data object I′.
17 . The method of claim 11 , further comprising generating the output video data object V′ by replacing a portion of at least one from the plurality of image frame data objects I with a corresponding portion of a replacement image frame data object I′, wherein the portion corresponds to a mouth region.
18 . The method of claim 11 , further comprising generating the output video data object V′ iteratively for each character present in the initial video data object V.
19 . A non-transitory computer readable medium, storing machine interpretable instruction sets, which, when executed by a processor, cause the processor to perform the steps of:
receiving an initial video data object V comprising a plurality of image frame data objects I;
extracting, from an initial audio data object A corresponding to initial video data object V, an initial set of expression/viseme codes corresponding to spectrogram segments in the initial audio data object A;
receiving a set of time-synchronized viseme parameters corresponding to target audio data object A′; and
processing the initial video data object V and the set of time-synchronized viseme parameters using a machine learning network to generate an output video data object V′, wherein initial mouth regions in the initial video data object V have been replaced with replacement mouth regions generated based on the set of time-synchronized viseme parameters;
wherein the initial video data object V is utilized to populate a viseme library of available visemes based on the initial set of expression/viseme codes; and
wherein the method further comprises comparing visemes identified in the target audio data object A′ to the viseme library of available visemes based on the initial set of expression/viseme codes to identify a subset of visemes not in the viseme library and requiring synthesis.