Synthetic speech generation
Disclosed are apparatuses, systems, and techniques that may use machine learning for generating artificial speech. The techniques include obtaining a synthetic embedding using learned embeddings associated with different speakers. At least one learned embedding may be generated using a multi-stage training of a machine learning model (MLM) with progressively increasing quality of training speech utterances. The techniques may further include using the MLM and the synthetic embedding to generate synthetic audio data.
1 . A method comprising:
obtaining a synthetic embedding using two or more learned embeddings associated with different speakers, at least one of the two or more learned embeddings being generated using a multi-stage training of a machine learning model (MLM) that was based at least on:
a first plurality of training utterances of a first quality during a first stage of the multi-stage training; and
a second plurality of training utterances of a second quality during a second stage of the multi-stage training, the second quality being higher than the first quality; and
generating audio data corresponding to a text representation based at least on the MLM processing the text representation and the synthetic embedding.
2 . The method of claim 1 , wherein the first quality of the first plurality of training utterances is characterized by a lower signal-to-noise ratio than the second quality of the second plurality of training utterances, wherein the first plurality of training utterances are associated with a first plurality of speakers and the second plurality of training utterances are associated with a second plurality of speakers, a number of the first plurality of speakers being larger than a number of the second plurality of speakers.
3 . The method of claim 1 , wherein the synthetic embedding is obtained, at least, by computing a weighted combination of the two or more learned embeddings.
4 . The method of claim 3 , wherein weights in the weighted combination of the two or more learned embeddings are selected randomly.
5 . The method of claim 1 , wherein the MLM comprises at least one transformer neural subnetwork with one or more attention layers.
6 . The method of claim 1 , wherein the MLM comprises:
a first subnetwork to associate units of the audio data with respective units of the text representation; and
a second subnetwork to determine durations of the units of the audio data.
7 . The method of claim 6 , wherein the first subnetwork and the second subnetwork comprise one or more convolutional layers and one or more fully connected layers.
8 . The method of claim 1 , wherein the text representation comprises a text embedding, and the text embedding is applied to the MLM in combination with the synthetic embedding.
9 . A method comprising:
obtaining a plurality of sets of training data, two or more sets of training data of the plurality of sets of training data being associated with a different audio quality (AQ) index characterizing audio quality of a corresponding set of training data, at least one set of training data of the plurality of sets of training data comprising:
a training input comprising a batch of text representations, and
a target output comprising a batch of audio data; and
training a machine learning model (MLM) using a plurality of training stages, at least one training stage of the plurality of training stages comprising applying the at least one set of training data to the MLM to generate learned embeddings corresponding to respective speakers associated with the at least one set of training data.
10 . The method of claim 9 , wherein at least one of:
the plurality of training stages are performed in an order of decreasing number of speakers associated with the at least one set of the training data; or
the plurality of training stages are performed in an order of increasing AQ index associated with the at least one set of training data.
11 . The method of claim 9 , wherein the MLM comprises at least one transformer neural subnetwork having one or more attention layers.
12 . The method of claim 9 , wherein one or more of the plurality of training stages comprise:
selecting a text representation from the batch of text representations of the training input for a corresponding training stage of the one or more training stages;
selecting audio data from the batch of audio data of the target output for the corresponding training stage;
training a first subnetwork of the MLM to associate units of the selected audio data with correct units of the selected text representation; and
training a second subnetwork of the MLM to determine duration of the units of the selected audio data.
13 . The method of claim 12 , wherein the units of the selected audio data comprise speech spectrograms.
14 . The method of claim 12 , wherein the first subnetwork and the second subnetwork comprise one or more convolutional layers of neurons and one or more fully connected layers of neurons.
15 . The method of claim 14 , wherein the one or more training stages of the plurality of training stages further comprise:
obtaining an output of the MLM comprising synthetic audio data generated for the selected text representation, the target speaker ID, and the embedding for the target speaker; and modifying the embedding for the target speaker based on a difference between the synthetic audio data and the selected audio data.
16 . The method of claim 9 , wherein one or more training stages of the plurality of training stages comprise:
selecting a text representation from the batch of text representations of the training input for a corresponding training stage of the one or more training stages;
selecting audio data from the batch of audio data of the target output for the corresponding training stage;
obtaining a target speaker identification (ID) identifying a target speaker associated with the selected audio data; and
applying, to the MLM, at least:
the selected text representation,
the target speaker ID, and
an embedding for the target speaker.
17 . The method of claim 16 , wherein the one or more training stages of the plurality of training stages further comprise:
obtaining an output of the MLM comprising synthetic audio data generated for the selected text representation, the target speaker ID, and the embedding for the target speaker; and modifying parameters of the MLM based on a difference between the synthetic audio data and the selected audio data.
18 . A system comprising:
one or more processing units to cause presentation of synthetic speech generated based at least on one or more machine learning models (MLMs) processing a synthetic embedding and an associated textual representation, the synthetic embedding generated based at least on combining two or more learned embeddings corresponding to two or more different speakers, wherein at least one MLM of the one or more MLMs is trained using a multi-stage training process where respective stages include different audio quality (AQ) indexes associated with respective sets of training data corresponding to the respective stage.
19 . The system of claim 18 , wherein the system is comprised in at least one of:
an in-vehicle infotainment system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting at least one of virtual reality content, mixed reality content, or augmented reality content;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.