Computer-implemented method and computer system for configuring a pretrained text to music AI model and related methods
The method involves configuring a pretrained text to music AI model that includes a neural network implementing a diffusion model. The process includes receiving audio sample data corresponding to a specific audio concept, generating a concept identifier token based on the audio sample data, adapting a loss function of the diffusion model based on the concept identifier token, selecting pivotal parameters in weight matrices in a self-attention layer of the neural network of the AI model based on the audio sample data, and further training the pivotal parameters of the AI model, to optimize the AI model for the specific audio concept.
1. A computer-implemented method for configuring a pretrained text to music artificial intelligence (AI) model that includes a neural network implementing a diffusion model, the method comprising:
receiving audio sample data corresponding to a specific audio concept;
generating at least one concept identifier token based on the audio sample data, wherein the at least one concept identifier token represents unique characteristics of the audio sample data;
adapting a loss function of the diffusion model based on the at least one concept identifier token;
selecting pivotal parameters in weight matrices in a self-attention layer of the neural network of the AI model based on the audio sample data; and
further training the pivotal parameters of the AI model, to thereby optimize the AI model for the specific audio concept, whereby the diffusion model is able to generate music corresponding to the specific audio concept.
2. The method of claim 1 , wherein the specific audio concept is the style of a specified artist.
3. The method of claim 1 , wherein the specific audio concept is the sound of a specified musical instrument.
4. The method of claim 1 , wherein the step of selecting pivotal parameters comprises:
initializing a trainable mask which has the same shape as the self-attention layer;
elementwise multiplying the trainable mask with parameters of the self-attention layer during calculation for the neural network to derive a refined mask form the trainable mask; and
selecting, as the pivotal parameters, subset of the parameters having a high variation between the trainable mask and the refined mask.
5. The method of claim 4 , wherein the subset comprises a predetermined percentage of the parameters.
6. The method of claim 4 , wherein the subset comprises a predetermined number of the parameters.
7. The method of claim 1 , wherein the at least one concept identifier token comprises two or more concept identifier tokens.
8. The method of claim 1 , wherein further training the pivotal parameters of the AI model, to thereby optimize the AI model for the specific audio concept comprises training only the pivotal parameters.
9. The method of claim 1 , wherein the specific concept is at least one of a music genre, an artist's style, and a musical instrument.
10. A computer system for configuring a pretrained text to music artificial intelligence (AI) model that includes a neural network implementing a diffusion model, the method comprising:
at least one computer hardware processor; and
at least one memory operatively couple to the at least one computer hardware and having computer executable instructions stored therein which, when executed by the at least one computer processor, cause the at least one computer processor to carry out a method comprising:
receiving audio sample data corresponding to a specific audio concept;
generating at least one concept identifier token based on the audio sample data, wherein the at least one concept identifier token represents unique characteristics of the audio sample data;
adapting a loss function of the diffusion model based on the at least one concept identifier token;
selecting pivotal parameters in weight matrices in a self-attention layer of the neural network of the AI model based on the audio sample data; and
further training the pivotal parameters of the AI model, to thereby optimize the AI model for the specific audio concept, whereby the diffusion model is able to generate music corresponding to the specific audio concept.
11. The system of claim 10 , wherein the specific audio concept is the style of a specified artist.
12. The system of claim 10 , wherein the specific audio concept is the sound of a specified musical instrument.
13. The method of claim 10 , wherein the step of selecting pivotal parameters comprises:
initializing a trainable mask which has the same shape as the self-attention layer;
elementwise multiplying the trainable mask with parameters of the self-attention layer during calculation for the neural network to derive a refined mask form the trainable mask; and
selecting, as the pivotal parameters, subset of the parameters having a high variation between the trainable mask and the refined mask.
14. The system of claim 13 , wherein the subset comprises a predetermined percentage of the parameters.
15. The system of claim 13 , wherein the subset comprises a predetermined number of the parameters.
16. The system of claim 10 , wherein the at least one concept identifier token comprises two or more concept identifier tokens.
17. The system of claim 10 , wherein further training the pivotal parameters of the AI model, to thereby optimize the AI model for the specific audio concept comprises training only the pivotal parameters.
18. The system of claim 10 , wherein the specific concept is at least one of a music genre, an artist's style, and a musical instrument.