Cross-speaker style transfer speech synthesis
This disclosure provides methods and apparatuses for training an acoustic model which is for implementing cross-speaker style transfer and comprises at least a style encoder. Training data may be obtained, which comprises a text, a speaker ID, a style ID and acoustic features corresponding to a reference audio. A reference embedding vector may be generated, through the style encoder, based on the acoustic features. Adversarial training may be performed to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information. A style embedding vector may be generated, through the style encoder, based at least on the reference embedding vector being performed the adversarial training. Predicted acoustic features may be generated based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.
1 . A method for training an acoustic model, the acoustic model being for implementing cross-speaker style transfer and comprising at least a style encoder comprising a convolutional neural network (CNN) and a long short-term memory (LSTM) network, the method comprising:
obtaining training data, the training data comprising a text, a speaker identity (ID), a style ID and acoustic features corresponding to a reference audio;
generating, through the style encoder comprising a convolutional neural network (CNN) and a long short-term memory (LSTM) network, a reference embedding vector of at least 128 dimensions based on the acoustic features;
performing adversarial training to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information, wherein the adversarial training comprises:
generating, through a style classifier comprising multiple neural network layers, a style classification result comprising probability distributions over a plurality of style categories for the reference embedding vector;
performing gradient reversal processing to the reference embedding vector by multiplying gradients computed during backpropagation by a negative scaling factor;
generating, through a speaker classifier comprising multiple neural network layers, a speaker classification result comprising probability distributions over a plurality of speaker identities for the reference embedding vector being performed the gradient reversal processing; and
calculating a gradient back-propagation factor through a loss function by computing partial derivatives across the multiple neural network layers of the style classifier and the speaker classifier, the loss function being based at least on a comparison result between the style classification result and the style ID and a comparison result between the speaker classification result and the speaker ID;
generating, through the style encoder, a style embedding vector based at least on the reference embedding vector being performed the adversarial training; and
generating predicted acoustic features based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.
2 . The method of claim 1 , wherein the generating the reference embedding vector comprises:
generating the reference embedding vector based on the acoustic features through a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network in the style encoder.
3 . The method of claim 1 , wherein
the adversarial training is performed by a Domain Adversarial Training (DAT) module.
4 . The method of claim 1 , wherein the generating a style embedding vector comprises:
generating, through a full connection layer in the style encoder, the style embedding vector based at least on the reference embedding vector being performed the adversarial training, or based at least on the reference embedding vector being performed the adversarial training and the style ID.
5 . The method of claim 4 , wherein the generating a style embedding vector comprises:
generating, through a second full connection layer in the style encoder, the style embedding vector based at least on the style ID, or based at least on the style ID and the speaker ID.
6 . The method of claim 1 , wherein
the style encoder is a Variational Auto Encoder (VAE) or a Gaussian Mixture Variational Auto Encoder (GMVAE).
7 . The method of claim 1 , wherein
the style embedding vector corresponds to a prior distribution or a posterior distribution of a latent variable having Gaussian distribution or Gaussian mixture distribution.
8 . The method of claim 1 , further comprising:
obtaining a plurality of style embedding vectors corresponding to a plurality of style IDs respectively, or obtaining a plurality of style embedding vectors corresponding to a plurality of combinations of style ID and speaker ID respectively, through training the acoustic model with a plurality of training data.
9 . The method of claim 1 , further comprising:
encoding the text into the state sequence through a text encoder in the acoustic model; and
generating the speaker embedding vector through a speaker look up table (LUT) in the acoustic model, and
the generating predicted acoustic features comprises:
extending the state sequence with the speaker embedding vector and the style embedding vector;
generating, through an attention module in the acoustic model, a context vector based at least on the extended state sequence; and
generating, through a decoder in the acoustic model, the predicted acoustic features based at least on the context vector.
10 . The method of claim 1 , further comprising: during applying the acoustic model,
receiving an input, the input comprising a target text, a target speaker ID, and a target style reference audio and/or a target style ID;
generating, through the style encoder, a style embedding vector based at least on acoustic features of the target style reference audio and/or the target style ID; and
generating acoustic features based at least on the target text, the target speaker ID and the style embedding vector.
11 . The method of claim 10 , wherein
the input further comprises a reference speaker ID, and
the generating a style embedding vector is further based on the reference speaker ID.
12 . The method of claim 1 , further comprising: during applying the acoustic model,
receiving an input, the input comprising a target text, a target speaker ID, and a target style ID;
selecting, through the style encoder, a style embedding vector from a plurality of predetermined candidate style embedding vectors based at least on the target style ID; and
generating acoustic features based at least on the target text, the target speaker ID and the style embedding vector.
13 . At least one non-transitory machine-readable medium including instructions for training an acoustic model for implementing cross-speaker style transfer using at least a style encoder comprising a convolutional neural network (CNN) and a long short-term memory (LSTM) network that, when executed by at least one processor, cause the at least one processor to perform operations to:
obtain training data, the training data comprising a text, a speaker identity (ID), a style ID and acoustic features corresponding to a reference audio;
generate, through the style encoder comprising a convolutional neural network (CNN) and a long short-term memory (LSTM) network, a reference embedding vector of at least 128 dimensions based on the acoustic features;
perform adversarial training to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information, wherein the adversarial training comprises:
generating, through a style classifier comprising multiple neural network layers, a style classification result comprising probability distributions over a plurality of style categories for the reference embedding vector;
performing gradient reversal processing to the reference embedding vector by multiplying gradients computed during backpropagation by a negative scaling factor;
generating, through a speaker classifier comprising multiple neural network layers, a speaker classification result comprising probability distributions over a plurality of speaker identities for the reference embedding vector being performed the gradient reversal processing; and
calculating a gradient back-propagation factor through a loss function by computing partial derivatives across the multiple neural network layers of the style classifier and the speaker classifier, the loss function being based at least on a comparison result between the style classification result and the style ID and a comparison result between the speaker classification result and the speaker ID;
generate, through the style encoder, a style embedding vector based at least on the reference embedding vector being performed the adversarial training; and
generate predicted acoustic features based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.
14 . An apparatus for training an acoustic model, the acoustic model being for implementing cross-speaker style transfer and comprising at least a style encoder comprising a convolutional neural network (CNN) and a long short-term memory (LSTM) network, the apparatus comprising:
at least one processor; and
a memory storing computer-executable instructions that, when executed, cause the at least one processor to:
obtain training data, the training data comprising a text, a speaker identity (ID), a style ID and acoustic features corresponding to a reference audio,
generate, through the style encoder comprising a convolutional neural network (CNN) and a long short-term memory (LSTM) network, a reference embedding vector of at least 128 dimensions based on the acoustic features,
perform adversarial training to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information, wherein the adversarial training comprises:
generating, through a style classifier comprising multiple neural network layers, a style classification result comprising probability distributions over a plurality of style categories for the reference embedding vector;
performing gradient reversal processing to the reference embedding vector by multiplying gradients computed during backpropagation by a negative scaling factor;
generating, through a speaker classifier comprising multiple neural network layers, a speaker classification result comprising probability distributions over a plurality of speaker identities for the reference embedding vector being performed the gradient reversal processing; and
calculating a gradient back-propagation factor through a loss function by computing partial derivatives across the multiple neural network layers of the style classifier and the speaker classifier, the loss function being based at least on a comparison result between the style classification result and the style ID and a comparison result between the speaker classification result and the speaker ID;
generate, through the style encoder, a style embedding vector based at least on the reference embedding vector being performed the adversarial training, and
generate predicted acoustic features based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.
15 . The apparatus of claim 14 , the instructions to generate the reference embedding vector further comprising instructions to generate the reference embedding vector based on the acoustic features through a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network in the style encoder.
16 . The apparatus of claim 14 , wherein the adversarial training is performed by a Domain Adversarial Training (DAT) module.
17 . The apparatus of claim 14 , the instructions to generate a style embedding vector further comprising instructions to generate, through a full connection layer in the style encoder, the style embedding vector based at least on the reference embedding vector being performed the adversarial training, or based at least on the reference embedding vector being performed the adversarial training and the style ID.
18 . The apparatus of claim 17 , the instructions to generate a style embedding vector further comprising instructions to generate, through a second full connection layer in the style encoder, the style embedding vector based at least on the style ID, or based at least on the style ID and the speaker ID.
19 . The at least one non-transitory machine-readable medium of claim 13 , the instructions to generate the reference embedding vector further comprising instructions to generate the reference embedding vector based on the acoustic features through a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network in the style encoder.
20 . The at least one non-transitory machine-readable medium of claim 13 , wherein the adversarial training is performed by a Domain Adversarial Training (DAT) module.