Artificial intelligence music generation model and method for configuring the same
The present disclosure provides a method for configuring a learning model for music generation and the corresponding learning model. The method includes training a masked autoencoder with training data comprising a combination of a reconstruction loss over time and frequency domains and a patch-based adversarial objective operating at different resolutions. An omnidirectional latent diffusion model is trained based on music data represented in a latent space to obtain a pretrained diffusion model. The pretrained diffusion model is fine-tuned based on text-guided music generation, bidirectional music in-painting, and unidirectional music continuation. The method enables high-fidelity music generation conditioned on text or music representations while maintaining computational efficiency.
1. A method for configuring a learning model for music generation, the method comprising:
providing a masked autoencoder which executes on a computing device;
providing an omnidirectional latent diffusion model which executes on a computing device, and which is operatively coupled to the masked autoencoder to process latent embeddings produced by the masked autoencoder;
training the masked autoencoder with training data, the training data including a combination of a reconstruction loss over time and frequency domains, and a patch-based adversarial objective operating at different resolutions, including processing of first training data with the masked autoencoder, applying a first loss function to the results of the processing of the first training data by the masked autoencoder, and adjusting parameters of the masked autoencoder in accordance with the loss function;
configuring a pretrained diffusion model by training the omnidirectional latent diffusion model based on music data represented in a latent space to obtain a pretrained diffusion model, including processing of second training data with the omnidirectional latent diffusion model, applying a second loss function to the results of the processing of the second training data by the omnidirectional latent diffusion model, and adjusting parameters of the omnidirectional latent diffusion model in accordance with the loss function;
fine-tuning the pretrained diffusion model based on text-guided music generation;
fine-tuning the pretrained diffusion model based on bidirectional music in-painting; and
fine-tuning the pretrained diffusion model based on unidirectional music continuation.
2. The method of claim 1 , wherein a data masking percentage of the masked autoencoder is 5 percent.
3. The method of claim 1 , wherein fine-tuning the pretrained diffusion model based on text-guided music generation includes a bidirectional mode and a unidirectional mode, wherein the bidirectional mode allows the latent embeddings to attend to one another during a denoising process, and wherein the unidirectional mode restricts the latent embeddings to attend solely to previous time counterparts thereof.
4. The method of claim 1 , wherein fine-tuning the pretrained diffusion model based on bidirectional music in-painting comprises simulating a music inpainting process by randomly generating audio masks and applying the audio mask to obtain corresponding masked audio, wherein the masked audio serves as conditional in-context learning inputs into the diffusion model in the process of fine-tuning the pretrained diffusion model.
5. The method of claim 1 , wherein fine-tuning the pretrained diffusion model based on unidirectional music continuation comprises simulating a music continuation process through the random generation of exclusive right-only masks.
6. The method of claim 1 , wherein the omnidirectional latent diffusion model includes at least one convolutional block and at least one transformer block.
7. The method of claim 6 , wherein the at least one convolutional block includes causal padding in a unidirectional mode to restrict the latent embeddings to attend solely to previous time counterparts thereof.
8. A system for music generation, comprising:
a masked autoencoder, executed on a computing device and trained with training data including a combination of a reconstruction loss over time and frequency domains, and a patch-based adversarial objective operating at different resolutions, wherein the training of the masked autoencoder includes processing of first training data with the masked autoencoder, applying a first loss function to the results to the processing of the first training data by the masked autoencoder, and adjusting parameters of the masked autoencoder in accordance with the loss function;
a pretrained omnidirectional latent diffusion model operatively coupled to the masked autoencoder to process latent embeddings produced by the masked autoencoder and which is trained based on music data represented in a latent space to obtain a pretrained diffusion model, wherein the training of the pretrained omnidirectional latent diffusion model includes processing of second training data with an omnidirectional latent diffusion model, applying a second loss function to the results of the processing of the second training data by the omnidirectional latent diffusion model, and adjusting parameters of the omnidirectional latent diffusion model in accordance with the loss function; and
wherein the pretrained omnidirectional latent diffusion model is fine-tuned based on text-guided music generation, bidirectional music in-painting, and unidirectional music continuation.
9. The system of claim 8 , wherein a masking percentage of the masked autoencoder is 5 percent.
10. The system of claim 8 , wherein fine-tuning the pretrained diffusion model based on text-guided music generation includes a bidirectional mode and a unidirectional mode, wherein the bidirectional mode allows the latent embeddings to attend to one another during a denoising process, and wherein the unidirectional mode restricts the latent embeddings to attend solely to previous time counterparts thereof.
11. The system of claim 8 , wherein fine-tuning the pretrained diffusion model based on bidirectional music in-painting comprises simulating a music inpainting process by randomly generating audio masks and applying the audio masks to obtain corresponding masked audio into the diffusion model in the process of fine-tuning the pretrained diffusion model.
12. The system of claim 11 , wherein the masked audio serves as conditional in-context learning inputs.
13. The system of claim 8 , wherein fine-tuning the pretrained diffusion model based on unidirectional music continuation comprises simulating a music continuation process through random generation of exclusive right-only masks.
14. The system of claim 8 , wherein the pretrained omnidirectional latent diffusion model includes at least one convolutional block and at least one transformer block, and wherein the at least one convolutional block includes causal padding in a unidirectional mode to restrict latent embeddings to attend solely to their previous time counterparts thereof.