Audio generation methods and system
A method of generating audio assets, comprising the steps of: receiving an input multi-layered audio asset comprising a plurality of audio layers, generating an input multi-channel image, wherein each channel of the input multi-channel image comprises an input image representative of one of the audio layers, training a generative model on the input multi-channel image and implementing the trained generative model to generate an output multi-channel image, wherein each channel of the output multi-channel image comprises an output image representative of an output audio layer, and generating an output multi-layered audio asset based on a combination of output audio layers derived from the output images.
1. A method of generating audio assets, comprising the steps of:
receiving an input multi-layered audio asset comprising a plurality of audio layers,
converting each audio layer into an input graphical representation,
generating an input multi-channel image by stacking each input graphical representation in a separate channel of the multi-channel image,
training a generative model on the input multi-channel image and implementing the trained generative model to generate an output multi-channel image, wherein each channel of the output multi-channel image comprises an output graphical representation of an output audio layer,
separating the output graphical representations from each output multi-channel image, and
generating an output multi-layered audio asset based on a combination of output audio layers derived from the output graphical representations.
2. The method according to claim 1 , wherein the step of generating an output multi-layered audio asset comprises arranging the plurality of output audio layers with a time delay between each of the output audio layers.
3. The method according to claim 2 , wherein the step of receiving an input multi-layered audio asset comprises determining the temporal arrangement of the audio layers, and
the time delay between each of the plurality of output audio layers is configured according to the determined temporal arrangement of the audio layers.
4. The method according to claim 1 , wherein the step of receiving an input multi-layered audio asset comprises labelling the sequence of audio layers in the input multi-layered audio asset, and
the step of generating an output multi-layered audio asset comprises arranging the output audio layers according to the labelled sequence of the input multi-layered audio asset.
5. The method according to claim 1 , wherein the generative model is a single-image generative model comprising a generative adversarial network, GAN, having a generator and a patch discriminator.
6. The method according to claim 1 , wherein each input image is a spectrogram of a respective audio layer, the input multi-channel image is an input multi-channel spectrogram and the output multi-channel image is an output multi-channel spectrogram.
7. The method according to claim 6 , wherein the step of generating an input multi-channel image comprises separating the audio layers in the input multi-channel image and performing a Fourier transform on each audio layer to generate a spectrogram of the respective audio layer.
8. A method according to claim 1 , wherein the step of receiving an input multi-layered audio asset comprises receiving, from a video game environment, video game information, and the step of generating the output multi-channel images comprises feeding the video game information into the single-image generative model such that the output multi-channel image is influenced by the video game information.
9. A method according to claim 1 , further comprising the step of storing the trained generative model on a memory, configured to be accessed to generate further audio assets.
10. A method according to claim 1 , wherein the step of receiving an input multi-layered audio asset comprises receiving a second input multi-layered audio asset having second input layers, and the input multi-channel image further comprises images representative of the second input layers.
11. A computer program comprising computer-implemented instructions that, when run on a computer, cause the computer to implement a method of generating audio assets, comprising the steps of:
receiving an input multi-layered audio asset comprising a plurality of audio layers,
converting each audio layer into an input graphical representation,
generating an input multi-channel image by stacking each input graphical representation in a separate channel of the multi-channel image,
training a generative model on the input multi-channel image and implementing the trained generative model to generate an output multi-channel image, wherein each channel of the output multi-channel image comprises an output graphical representation of an output audio layer,
separating the output graphical representations from each output multi-channel image, and
generating an output multi-layered audio asset based on a combination of output audio layers derived from the output graphical representation.
12. A system for generating audio assets, the system comprising:
an asset input unit configured to receive an input multi-layered audio asset comprising a plurality of audio layers, convert each audio layer into an input graphical representation, and generate an input multi-channel image by stacking each input graphical representation in a separate channel of the multi-channel image, and
an image generation unit configured to implement a generative model to generate one or more output multi-channel images based on the input multi-channel image, each channel of the output multi-channel image comprising an output graphical representation of an output audio layer, and
an asset output unit configured to separate the output graphical representations from each output multi-channel image and generate an output multi-layered audio asset based on a combination of output audio layers derived from the output graphical representations.
13. A system according to claim 12 , further comprising a transform unit configured to perform Fourier transform operations and inverse Fourier transform operations to convert between audio and graphical files, and wherein
the asset input unit is configured to access the transform unit to convert each input audio layer into an input graphical representation, and
the asset output unit is configured to access the transform unit to convert each output graphical representation into an output audio layer.
14. A system according to any of claim 12 , further comprising a video game data processing unit, configured to process video game information derived from or relating to a video game environment and feed through to one or more of the asset input unit, the image generation unit and the asset output unit, and the image generation unit is configured to implement the generative model based at least in part on the video game information.
15. A system according to claim 12 , configured to store the generative model on the memory.