Methods and systems of text-conditioned audio-visual speech generation with multi-modal latent diffusion models
Methods, systems, and computer programs are presented for audio-visual speech generation with multi-modal latent diffusion models. One method includes encoding raw audio signals and video frames into respective latent spaces using audio and visual autoencoders. A text transcript is processed into phoneme sequences using a text transcript processor. The audio and visual latent spaces are conditioned using the text transcript and a conditioning variable. Joint distributions of the visual and audio latent spaces, text transcript, and conditioning variable are learned using a multi-modal latent diffusion model. The model adds noise to the latent audio-visual representations and predicts the noise through denoising neural networks. An inverted diffusion process is utilized to generate diverse speech content and speaker characteristics, resulting in realistic audio-visual speech. The technology presented provides a novel approach to conditional speech generation with potential applications in speech synthesis, voice conversion, and speech recognition.
1 . A computer-implemented method, for conditional audio-visual speech generation, comprising:
encoding raw audio signals into an audio latent space using an audio autoencoder;
encoding raw video frames into a visual latent space using a visual autoencoder;
processing text transcripts associated with the raw audio signals into phoneme sequences using a text transcript processor;
conditioning the audio latent space and the visual latent space using the text transcripts and a conditioning variable;
learning a generative model based on joint distributions of the visual latent space, the audio latent space, the text transcripts, and the conditioning variable using a multi-modal latent diffusion model by adding noise to latent audio-visual representations of the audio latent space and the visual latent space and predicting said noise through denoising neural networks;
receiving a text input and a reference video speech of a speaker; and
generating, by the generative model, an audio-visual output based on the text input and the reference video speech of the speaker, wherein an audio signal of the audio-visual output contains speech based on the text input and a video signal of the audio-visual output includes a video of the speaker coordinated with the audio signal.
2 . The computer-implemented method as claimed in claim 1 , wherein the conditioning variable includes parameters specifying emotional expression in the raw audio signals and raw video frames.
3 . The computer-implemented method as claimed in claim 2 , wherein the conditioning variable includes parameters related to gender of the speaker, allowing the generated audio-visual output to exhibit gender-specific characteristics.
4 . The computer-implemented method as claimed in claim 1 , wherein the audio autoencoder employs convolutional neural networks (CNNs) to transform the raw audio signals into mel-spectrogram representations before encoding into the audio latent space.
5 . The computer-implemented method as claimed in claim 1 , wherein the visual autoencoder utilizes 3D convolutional networks to encode the raw video frames into the visual latent space.
6 . The computer-implemented method as claimed in claim 1 , wherein a style encoder encodes a reference audio-visual sample to extract specific style characteristics to be applied to the generated audio-visual output.
7 . The computer-implemented method as claimed in claim 1 , wherein the denoising neural networks utilize cross-modal connections to align audio and visual modalities.
8 . The computer-implemented method as claimed in claim 1 , wherein the inverted diffusion process includes generating diverse speech content by manipulating the audio latent space and the visual latent space representations based on the text transcripts and additional conditioning variables.
9 . The computer-implemented method as claimed in claim 1 , wherein the inverted diffusion process includes adjusting noise levels based on a complexity of desired speech content, ensuring realistic generation in varied scenarios.
10 . The computer-implemented method as claimed in claim 1 , wherein the denoising neural networks employ attention mechanisms to focus on specific phonemes and facial expressions, enhancing an accuracy of noise prediction and speech generation.
11 . The computer-implemented method as claimed in claim 1 , wherein the joint distributions are learned iteratively, refining the generation process based on user feedback and preferences for specific speaker characteristics and speech styles.
12 . A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising:
encoding raw audio signals into an audio latent space using an audio autoencoder;
encoding raw video frames into a visual latent space using a visual autoencoder;
processing text transcripts associated with the raw audio signals into phoneme sequences using a text transcript processor;
conditioning the audio latent space and the visual latent space using the text transcripts and a conditioning variable;
learning a generative model based on joint distributions of the visual latent space, the audio latent space, the text transcripts transcript, and the conditioning variable using a multi-modal latent diffusion model by adding noise to latent audio-visual representations of the audio latent space and the visual latent space and predicting said noise through denoising neural networks;
receiving a text input and a reference video speech of a speaker; and
generating, by the generative model, an audio-visual output based on the text input and the reference video speech of the speaker, wherein an audio signal of the audio-visual output contains speech based on the text input and a video signal of the audio-visual output includes a video of the speaker coordinated with the audio signal.
13 . The system as recited in claim 12 , wherein the conditioning variable includes parameters specifying emotional expression in the raw audio signals and raw video frames.
14 . The system as recited in claim 13 , wherein the conditioning variable includes parameters related to gender of the speaker, allowing the generated audio-visual output to exhibit gender-specific characteristics.
15 . The system as recited in claim 12 , wherein the audio autoencoder employs convolutional neural networks (CNNs) to transform the raw audio signals into mel-spectrogram representations before encoding into the audio latent space.
16 . The system as recited in claim 12 , wherein the visual autoencoder utilizes 3D convolutional networks to encode the raw video frames into the visual latent space.
17 . A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
encoding raw audio signals into an audio latent space using an audio autoencoder;
encoding raw video frames into a visual latent space using a visual autoencoder;
processing text transcripts associated with the raw audio signals into phoneme sequences using a text transcript processor;
conditioning the audio latent space and the visual latent space using the text transcripts and a conditioning variable;
learning a generative model based on joint distributions of the visual latent space, the audio latent space, the text transcripts, and the conditioning variable using a multi-modal latent diffusion model by adding noise to latent audio-visual representations of the audio latent space and the visual latent space and predicting said noise through denoising neural networks;
receiving a text input and a reference video speech of a speaker; and
generating, by the generative model, an audio-visual output based on the text input and the reference video speech of the speaker, wherein an audio signal of the audio-visual output contains speech based on the text input and a video signal of the audio-visual output includes a video of the speaker coordinated with the audio signal.
18 . The non-transitory machine-readable storage medium as recited in claim 17 , wherein the conditioning variable includes parameters specifying emotional expression in the raw audio signals and raw video frames.
19 . The non-transitory machine-readable storage medium as recited in claim 18 , wherein the conditioning variable includes parameters related to gender of the speaker, allowing the generated audio-visual output to exhibit gender-specific characteristics.