Audio generation using generative artificial intelligence model
A method. The method including receiving a prompt describing desired characteristics of audio. The method further including generating, using a set of machine learning models and based on the prompt, a latent space representation of the audio at a latent rate less than 40 Hz. The method further including generating, using the set of machine learning models and the latent space representation of the audio, an audio file at an output rate greater than the latent rate. The audio file including the audio based on the latent space representation of the audio. The audio having a length greater than 90 seconds.
1 . A system comprising:
one or more storage media storing instructions; and
one or more processors configured to execute the instructions to cause the system to:
receive a prompt describing desired characteristics of audio;
generate, using a set of machine learning models and based on the prompt, a latent space representation of the audio at a latent rate less than 40 Hz; and
generate, using the set of machine learning models and the latent space representation of the audio, an audio file at an output rate of at least 40 kHz and including the audio based on the latent space representation of the audio, the audio having a length greater than 90 seconds.
2 . The system of claim 1 , wherein the latent rate is between 20 Hz and 25 Hz.
3 . The system of claim 1 , wherein the set of machine learning models includes a text embedding model, a latent diffusion model, and a decoder model.
4 . The system of claim 3 , wherein generating using the set of machine learning models comprises:
generating, using the text embedding model, a second latent space representation of the prompt;
generating, using the latent diffusion model, the latent space representation of the audio using the second latent space representation of the prompt; and
generating, by at least using the decoder model and the latent space representation of the audio, the audio.
5 . The system of claim 1 , wherein the audio represented by the audio file has a structure, the structure including at least two of the following: an intro, a verse, a pre-chorus, a chorus, a bridge, a hook, a refrain, an instrumental break, or an outro.
6 . The system of claim 1 , wherein the set of machine learning models includes a decoder model trained to generate the audio at the output rate using the latent space representation of the audio at the latent rate of 21.5 Hz.
7 . The system of claim 1 , wherein the prompt includes at least one of: text, audio, image, or video.
8 . The system of claim 1 , wherein the audio represented by the audio file includes at least one of: music, voice, an instrumental, or a sound effect.
9 . The system of claim 1 , wherein the audio file includes one or more channels that are represented independently of the other channels.
10 . The system of claim 1 , wherein the set of machine learning models includes a diffusion model trained using structured audio including at least two of the following: an intro, a verse, a pre-chorus, a chorus, a bridge, a hook, a refrain, an instrumental break, or an outro.
11 . A computer-implemented method comprising:
receiving a prompt describing desired characteristics of audio;
generating, using a set of machine learning models and based on the prompt, a latent space representation of the audio at a latent rate less than 40 Hz; and
generating, using the set of machine learning models and the latent space representation of the audio, an audio file at an output rate of at least 40 kHz and including the audio based on the latent space representation of the audio, the audio having a length greater than 90 seconds.
12 . The computer-implemented method of claim 11 , further comprising:
transmitting the audio file to a user device, the audio file including a first channel represented independently of a second channel, and the second channel represented independently of the first channel;
receiving, from the user device, a second prompt describing how the first channel of the audio file should be different;
generating, using the set of machine learning models and based on the second prompt, a second latent space representation of the audio at the latent rate; and
generating, using the set of machine learning models and the second latent space representation of the audio, a different first channel audio file at the output rate.
13 . The computer-implemented method of claim 11 , wherein all portions of the audio correspond uniquely to a respective portion of the latent space representation.
14 . The computer-implemented method of claim 11 , wherein the set of machine learning models includes a diffusion model trained using second audio with the length.
15 . The computer-implemented method of claim 11 , wherein the audio represented by the audio file has a structure, the structure including at least two of the following: an intro, a verse, a pre-chorus, a chorus, a bridge, a hook, a refrain, an instrumental break, or an outro.
16 . One or more non-transitory computer-readable storage media storing instructions that, upon execution executable by one or more processors of a system, cause the system to perform operations comprising:
receiving a prompt describing desired characteristics of audio;
generating, using a set of machine learning models and based on the prompt, a latent space representation of the audio at a latent rate less than 40 Hz; and
generating, using the set of machine learning models and the latent space representation of the audio, an audio file at an output rate of at least 40 kHz and including the audio based on the latent space representation of the audio, the audio having a length greater than 90 seconds.
17 . The non-transitory computer-readable storage medium of claim 16 , wherein the set of machine learning models includes a decoder model and was trained by operations comprising:
generating, using an encoder model and second audio having the output rate, a training latent space representation of the second audio and at the latent rate;
generating, using a previous configuration of the decoder model and the training latent space representation, a third audio at the output rate and representing the second audio;
comparing the second audio and the third audio; and
adjusting weights of at least one of the encoder model or the decoder model based on the comparing.
18 . The non-transitory computer-readable storage medium of claim 16 , wherein the latent rate is between 20 Hz and 25 Hz.
19 . The non-transitory computer-readable storage medium of claim 16 , wherein the set of machine learning models includes a diffusion model trained using structured audio with the length.
20 . The non-transitory computer-readable storage medium of claim 16 , wherein the audio represented by the audio file has a structure, the structure including at least two of the following: an intro, a verse, a pre-chorus, a chorus, a bridge, a hook, a refrain, an instrumental break, or an outro.