Neural network generative modeling to transform speech utterances and augment training data
Systems, methods, and devices for speech transformation and generating synthetic speech using deep generative models are disclosed. A method of the disclosure includes receiving input audio data comprising a plurality of iterations of a speech utterance from a plurality of speakers. The method includes generating an input spectrogram based on the input audio data and transmitting the input spectrogram to a neural network configured to generate an output spectrogram. The method includes receiving the output spectrogram from the neural network and, based on the output spectrogram, generating synthetic audio data comprising the speech utterance.
1. A method for generating synthetic speech data, the method comprising:
receiving input audio data comprising a speech utterance;
generating an input spectrogram based on the input audio data;
providing the input spectrogram to one or more strided convolutions of an encoder module of a neural network configured to process the input spectrogram to generate a resultant computation;
receiving an output spectrogram from a decoder module of the neural network configured to generate the output spectrogram based on the resultant computation; and
based on the output spectrogram, generating synthetic audio data comprising the speech utterance.
2. The method of claim 1 , wherein one or more of the input spectrogram and the output spectrogram comprises a two-dimensional audio spectrogram representation.
3. The method of claim 1 , wherein the speech utterance comprises one or more of a word, a phrase, a sentence, or a noise.
4. The method of claim 1 , wherein the input audio data comprises a plurality of iterations of the speech utterance from a plurality of speakers, and wherein the plurality of speakers comprises real-life speakers and synthetic speakers.
5. The method of claim 1 , further comprising transmitting the synthetic audio data comprising the speech utterance to a training neural network configured to be trained using the synthetic audio data.
6. The method of claim 1 , wherein the neural network comprises a modified neural network architecture comprising a convolutional layer in the encoder module and a deconvolutional layer in the decoder module.
7. The method of claim 6 , wherein the neural network is configured to undergo a training phase and a generation phase, wherein the training phase comprises learning a plurality of parameters for producing the output spectrogram and the generation phase comprises generating the output spectrogram by sampling from a latent layer.
8. The method of claim 1 , wherein the input audio data comprises a one-dimensional input audio signal.
9. The method of claim 1 , wherein providing the input spectrogram to the neural network comprises feeding the input spectrogram using a Mean Square Error loss function as a training criterion.
10. The method of claim 1 , wherein the decoder module of the neural network is configured to generate the output spectrogram by way of Griffin-Lim reconstruction.
11. The method of claim 1 , wherein one or more of the input spectrogram and the output spectrogram comprises a two-dimensional image comprising a time axis and a frequency axis.
12. A system for generating synthetic speech data, the system comprising:
a neural network configured to generate a two-dimensional audio spectrogram; and
non-transitory computer-readable storage media storing instructions that, when executed by one or more processors cause the one or more processors to:
receive input audio data comprising a plurality of iterations of a speech utterance from a plurality of speakers;
generate an input spectrogram based on the input audio data;
provide the input spectrogram to one or more strided convolutions of an encoder module of the neural network configured to process the input spectrogram to generate a resultant computation;
receive an output spectrogram from a decoder module of the neural network configured to generate the output spectrogram based on the resultant computation; and
based on the output spectrogram, generate synthetic audio data comprising the speech utterance.
13. The system of claim 12 , wherein the neural network comprises a modified neural network architecture based on a variational autoencoder model, and wherein the neural network comprises a convolutional layer used in the encoder module and a deconvolutional layer used in the decoder module.
14. The system of claim 12 , wherein the neural network comprises a workflow and the workflow comprises:
a training phase wherein the neural network functions as an autoencoder and is configured to learn a plurality of parameters for generating the output spectrogram; and
a generation phase wherein the neural network generates the output spectrogram.
15. The system of claim 14 , wherein the neural network undergoes the generation phase after the neural network has been trained and an encoding layer of the neural network has been removed.
16. The system of claim 12 , wherein the decoder module of the neural network is configured to generate the output spectrogram by sampling from a latent layer.
17. The system of claim 12 , wherein the non-transitory computer-readable storage media causes the one or more processors to generate the synthetic audio data by way of Griffin-Lim reconstruction to recover phase information.
18. The system of claim 12 , wherein the non-transitory computer-readable storage media causes the one or more processors to provide the input spectrogram to the one or more strided convolutions of the encoder module of the neural network with a Mean Square Error loss function as a training criterion, wherein errors are backpropagated using gradient descent optimization.
19. The system of claim 12 , wherein the neural network comprises a variational autoencoder architecture comprising a strided convolution, a fully connected convolution, and a strided deconvolution.
20. Non-transitory computer readable storage media storing instructions for execution by one or more processors, the instructions comprising:
receiving input audio data comprising a plurality of iterations of a speech utterance from a plurality of speakers;
generating a two-dimensional input audio spectrogram based on the input audio data;
providing the two-dimensional input audio spectrogram to one or more strided convolutions of an encoder module of a neural network configured to process the input spectrogram to generate a resultant computation;
receiving a two-dimensional output audio spectrogram from a decoder module of the neural network configured to generate the two-dimensional output audio spectrogram based on the resultant computation; and
based on the two-dimensional output audio spectrogram, generating synthetic audio data comprising the speech utterance.