Identity transfer models for generating audio/video content
Systems, devices, and methods are provided for training and/or inferencing using machine-learning models. In at least one embodiment, a user selects a source media (e.g., video or audio file) and a target identity. A content embedding may be extracted from the source media, and an identity embedding may be obtained for the target identity. The content embedding of the source media and the identity embedding of the target identity may be provided to a transfer model that generates synthesized media. For example, a user may select a song that is sung by a first artist and then select a second artist as the target identity to produce a cover of the song in the voice of the second artist.
1. A computer-implemented method, comprising:
obtaining a plurality of audio samples of songs associated with respective identities;
determining, based on the plurality of audio samples of songs and using a first encoder, multiple voice identity embeddings for the respective identities, including a first voice identity embedding associated with the first identity;
recording a first audio data sample from a user;
determining a second voice identity embedding for the user based on the first audio data;
storing, in an identity store, the multiple voice identity embeddings associated with the respective identities, including the first voice embedding for the first identity;
obtaining a request, from the user, to generate synthesized song audio, wherein the request specifies:
audio song content associated with a second identity; and
the first identity, wherein the first identity corresponds to the user, and the first voice identity embedding in the identity store is based on the first audio data;
determining a spectrogram of the audio song content;
determining, based on the spectrogram and using a second encoder, a content embedding associated with the audio song content; and
generating, using a second decoder corresponding to the second encoder and based on the first voice identity embedding and the content embedding, the synthesized song audio.
2. The computer-implemented method of claim 1 , further comprising:
training a first autoencoder comprising the first encoder and a first decoder, the first autoencoder trained to identify content-independent voice identities by at least:
obtaining first training audio data;
obtaining a first training spectrogram of the first training audio data;
splitting the first training audio data into a plurality of training segments;
selecting a first training segment of the plurality of training segments;
providing the first training segment as a training input to the first encoder to determine a first training voice identity embedding;
providing the first training voice identity embedding to the first decoder to determine a first training output;
computing a first loss based on a comparison between the first training output and a randomly selected training segment of the plurality of training segments; and
updating a first set of parameters of the first autoencoder based on the first loss.
3. The computer-implemented method of claim 2 , further comprising:
training a second autoencoder comprising the second encoder and the second decoder by at least:
applying a transformation to the first training audio data to produce a first transformed training audio data;
obtaining a first transformed training spectrogram of the first training audio data;
providing the first transformed training spectrogram to the second encoder to determine a first training content embedding;
providing the first training content embedding and the first training voice identity embedding to the second decoder to determine a second training output;
computing a second loss based on a comparison between the second training output and the first training spectrogram; and
updating a second set of parameters of the second autoencoder based on the second loss.
4. The computer-implemented method of claim 3 , wherein applying the transformation to the first training audio data to produce the first transformed training audio data comprises randomly changing pitch of the first training audio data.
5. A system, comprising:
one or more processors; and
memory storing executable instructions that, as a result of execution by the one or more processors, cause the system to:
obtain a plurality of audio samples associated with respective identities;
determine, based on the plurality of audio samples and using a first encoder, multiple voice identity embeddings for the respective identities, including a first voice identity embedding for a second identity;
store the multiple voice identity embeddings associated with the respective identities, including the first voice embedding for the second identity, in an identity store;
obtain a request, from a user, to generate synthesized audio song content, wherein the request specifies:
audio song content associated with a first identity; and
the second identity, wherein the second identity corresponds to the user, and the first voice identity embedding in the identity store is based on audio data recorded from the user;
determine a spectrogram of at least a portion of the audio song content;
determine, based on the spectrogram and using a second encoder, a content embedding associated with the audio song content;
obtain the first voice identity embedding associated with the second identity, wherein the first voice identity embedding is obtained from the identity store; and
generate, using a second decoder corresponding to the second encoder and based on the first voice identity embedding and the content embedding, the synthesized audio song content.
6. The system of claim 5 , wherein executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:
perform a contents rights check to determine whether the user has access to use the first voice identity embedding to generate the synthesized audio song content.
7. The system of claim 5 , wherein executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to train a first autoencoder comprising the first encoder and a first decoder, the first autoencoder trained to:
obtain first training audio data;
obtain a first training spectrogram of the first training audio data;
split the first training audio data into a plurality of training segments;
select a first training segment of the plurality of training segments;
provide the first training segment as a training input to the first encoder to determine a first training voice identity embedding;
provide the first training voice identity embedding to the first decoder to determine a first training output;
compute a first loss based on a comparison between the first training output and a randomly selected training segment of the plurality of training segments; and
update a first set of parameters of the first autoencoder based on the first loss.
8. The system of claim 5 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to train a second autoencoder comprising the second encoder and the second decoder, the second autoencoder trained to at least:
apply a transformation to the first training audio data to produce a first transformed training audio data;
obtain a first transformed training spectrogram of the first training audio data;
provide the first transformed training spectrogram to the second encoder to determine a first training content embedding;
provide the first training content embedding and the first training voice identity embedding to the second decoder to determine a second training output;
compute a second loss based on a comparison between the second training output and the first training spectrogram; and
update a second set of parameters of the second autoencoder based on the second loss.
9. The system of claim 5 , wherein the spectrogram is a mel spectrogram.
10. The system of claim 5 , wherein executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:
obtain a second request, from the user, to generate second synthesized audio song content, wherein the second request specifies:
second audio song content associated with a third identity; and
a fourth identity, wherein the identity store includes a third voice identity embedding for the fourth identity;
determine a second spectrogram of at least a portion of the second audio song content;
determine, based on the second spectrogram, a second content embedding associated with the second audio song content;
obtain the third voice identity embedding associated with the fourth identity, wherein the third voice identity embedding is obtained from the identity store; and
generate, based on the third voice identity embedding and the second content embedding, the second synthesized audio song content.
11. A computer-implemented method, comprising:
obtaining a plurality of audio samples associated with a second identity;
determining, based on the plurality of audio samples and using a first encoder, multiple voice identity embeddings for the respective identities, including a first voice identity embedding for the second identity;
storing the multiple voice identity embeddings associated with the respective identities, including the first voice embedding for the second identity, in an identity store;
obtaining a request, from a user, to generate synthesized audio content, wherein the request specifies:
audio song content associated with a first identity; and
the second identity, wherein the second identity corresponds to the user, and the first voice identity embedding in the identity store is based on audio data recorded from the user;
determining a spectrogram of at least a portion of the audio song content;
determining, based on the spectrogram and using a second encoder, a content embedding associated with the audio song content;
obtaining the first voice identity embedding associated with the second identity determined based on other audio song content associated with the second identity, wherein the first voice identity embedding is obtained from the identity store; and
generating, using a second decoder corresponding to the second encoder and based on the first voice identity embedding and the content embedding, the synthesized audio song content.
12. The method of claim 11 , further comprising:
performing a contents rights check to determine whether the user has access to use the first voice identity embedding to generate a synthesized audio content.
13. The method of claim 11 , further comprising training a first autoencoder comprising the first encoder and a first decoder, the first autoencoder trained to:
obtain first training audio song data;
obtain a first training spectrogram of the first training audio song data;
split the first training audio data into a plurality of training segments;
select a first training segment of the plurality of training segments;
provide the first training segment as a training input to the first encoder to determine a first training voice identity embedding;
provide the first training voice identity embedding to the first decoder to determine a first training output;
compute a first loss based on a comparison between the first training output and a randomly selected training segment of the plurality of training segments; and
update a first set of parameters of the first autoencoder based on the first loss.
14. The method of claim 11 , further comprising training a second autoencoder including the second encoder and the second decoder, the second autoencoder trained to at least:
apply a transformation to first training audio data to produce a first transformed training audio data;
obtain a first transformed training spectrogram of the first training audio data;
provide the first transformed training spectrogram to the second encoder to determine a first training content embedding;
provide the first training content embedding and the first training voice identity embedding to the second decoder to determine a second training output;
compute a second loss based on a comparison between the second training output and the first training spectrogram; and
update a second set of parameters of the second autoencoder based on the second loss.
15. The method of claim 11 , wherein the spectrogram is a mel spectrogram.
16. The method of claim 11 , further comprising:
obtaining a second request, from the user, to generate second synthesized audio song content, wherein the second request specifies:
second audio song content associated with a third identity; and
a fourth identity, wherein the identity store includes a third voice identity embedding for the fourth identity;
determining a second spectrogram of at least a portion of the second audio song content;
determining, based on the second spectrogram, a second content embedding associated with the second audio song content;
obtaining the third voice identity embedding associated with the fourth identity, wherein the third voice identity embedding is obtained from the identity store; and
generating, based on the third voice identity embedding and the second content embedding, the second synthesized audio song content.