Techniques for improved zero-shot voice conversion with a conditional disentangled sequential variational auto-encoder
A method, system, apparatus, and computer-readable medium for voice conversion using a conditional disentangled sequential variational auto-encoder (C-DSVAE) is provided. The method, performed by at least one processor, includes receiving input speech segments, encoding the input speech segments via a shared encoder to generate a speaker embedding and a content embedding, and encoding a posterior distribution of the speaker embedding via a speaker encoder and encoding a posterior distribution of the content embedding via a content encoder to obtain encoded results. The method further includes enabling a content bias, reshaping the content embedding using the content bias, and generating a reconstructed speech output based on the encoded results and the reshaped content embedding.
1 . A method for voice conversion using a conditional disentangled sequential variational auto-encoder (C-DSVAE), performed by at least one processor and comprising:
receiving input speech segments;
encoding the input speech segments via a shared encoder to generate a speaker embedding and a content embedding;
encoding a posterior distribution of the speaker embedding via a speaker encoder and encoding a posterior distribution of the content embedding via a content encoder to obtain encoded results;
enabling a content bias, and reshaping the content embedding by preserving phonetic information using the content bias sampled from the posterior distribution; and
generating a reconstructed speech output based on the encoded results and the reshaped content embedding,
wherein a total loss of the voice conversion is based on a reconstruction loss between the input speech segments and the reconstructed speech output, a prior distribution and the posterior distribution of the speaker embedding, and a Kullback-Leibler (KL)-Divergence between a conditional prior and the posterior distribution of the content embedding,
wherein the content bias is forced alignment, and the content bias is concatenated to prior modeling during a training process, and
wherein during the training process, the content embedding and a target embedding are concatenated when inferencing to obtain a voice conversion speech output, and the concatenated results of the content embedding and the target embedding are provided to a decoder to generate the reconstructed speech in a form of a spectrogram.
2 . The method of claim 1 , wherein the content bias is pseudo labels.
3 . The method of claim 1 , wherein the method is performed on voice cloning toolkit (VCTK) datasets.
4 . The method of claim 1 , wherein segments are randomly selected from the input speech segments for training the C-DSVAE.
5 . The method of claim 1 , further comprising converting the reconstructed speech output into a waveform.
6 . An apparatus for voice conversion using a conditional disentangled sequential variational auto-encoder (C-DSVAE), the apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
receiving code configured to cause the at least one processor to receive input speech segments;
first encoding code configured to cause the at least one processor to encode the input speech segments via a shared encoder to generate a speaker embedding and a content embedding;
second encoding code configured to cause the at least one processor to encode a posterior distribution of the speaker embedding via a speaker encoder and encode a posterior distribution of the content embedding via a content encoder to obtain encoded results;
enabling code configured to cause the at least one processor to enable a content bias, and reshape the content embedding by preserving phonetic information using the content bias sampled from the posterior distribution; and
generating code configured to cause the at least one processor to generate a reconstructed speech output based on the encoded results and the reshaped content embedding,
wherein a total loss of the voice conversion is based on a reconstruction loss between the input speech segments and the reconstructed speech output, a prior distribution and the posterior distribution of the speaker embedding, and a Kullback-Leibler (KL)-Divergence between a conditional prior and the posterior distribution of the content embedding, and
wherein the content bias is forced alignment, and the content bias is concatenated to prior modeling during a training process, and
wherein during the training process, the content embedding and a target embedding are concatenated when inferencing to obtain a voice conversion speech output, and the concatenated results of the content embedding and the target embedding are provided to a decoder to generate the reconstructed speech in a form of a spectrogram.
7 . The apparatus of claim 6 , wherein the content bias is pseudo labels.
8 . The apparatus of claim 6 , wherein the method is performed on voice cloning toolkit (VCTK) datasets.
9 . The apparatus of claim 6 , wherein segments are randomly selected from the input speech segments for training the C-DSVAE.
10 . The apparatus of claim 6 , further comprising converting code configured to cause the at least one processor to convert the reconstructed speech output into a waveform.
11 . A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by at least one processor of an apparatus for voice conversion using a conditional disentangled sequential variational auto-encoder (C-DSVAE) storing instructions that, cause the at least one processor to:
receive input speech segments;
encode the input speech segments via a shared encoder to generate a speaker embedding and a content embedding;
encode a posterior distribution of the speaker embedding via a speaker encoder and encode a posterior distribution of the content embedding via a content encoder to obtain encoded results;
enable a content bias, and reshape the content embedding by preserving phonetic information using the content bias sampled from the posterior distribution; and
generate a reconstructed speech output based on the encoded results and the reshaped content embedding to generate the reconstructed speech output,
wherein a total loss of the voice conversion is based on a reconstruction loss between the input speech segments and the reconstructed speech output, a prior distribution and the posterior distribution of the speaker embedding, and a Kullback-Leibler (KL)-Divergence between a conditional prior and the posterior distribution of the content embedding,
wherein the content bias is forced alignment, and the content bias is concatenated to prior modeling during a training process, and
wherein during the training process, the content embedding and a target embedding are concatenated when inferencing to obtain a voice conversion speech output, and the concatenated results of the content embedding and the target embedding are provided to a decoder to generate the reconstructed speech in a form of a spectrogram.
12 . The non-transitory computer-readable medium of claim 11 , wherein the content bias is pseudo labels.
13 . The non-transitory computer-readable medium of claim 11 , wherein the method is performed on voice cloning toolkit (VCTK) datasets.
14 . The non-transitory computer-readable medium of claim 11 , wherein segments are randomly selected from the input speech segments for training the C-DSVAE.