Generative adversarial network based audio restoration
View Patent ↗Mechanisms are provided for implementing a generative adversarial network (GAN) based restoration system. A first neural network of a generator of the GAN based restoration system is trained to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector. An original audio recording spectrogram is input to the trained generator, where the original audio recording spectrogram corresponds to an original audio recording and an input target damage vector. The trained generator processes the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector. A spectrogram inversion module converts the artificial audio recording spectrogram to an artificial audio recording waveform output.
1. A method, in a data processing system comprising a processor and a memory, the memory comprising instructions which are executed by the processor to specifically configure the processor to implement a generative adversarial network (GAN) based restoration system, the method comprising:
training, by the GAN based restoration system, a first neural network of a generator of the GAN based restoration system, to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector, thereby generating a trained generator;
inputting, by the GAN based restoration system to the trained generator, an original audio recording spectrogram corresponding to an original audio recording and an input target damage vector;
processing, by the trained generator, the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector; and
converting, by a spectrogram inversion module of the GAN based restoration system, the artificial audio recording spectrogram to an artificial audio recording waveform output.
2. The method of claim 1 , wherein the method further comprises:
training, by the GAN based restoration system, a second neural network of a discriminator of the GAN based restoration system, to discriminate between artificial audio spectrograms and true audio spectrograms, and thereby generate a trained discriminator.
3. The method of claim 2 , wherein training the first neural network comprises training the first neural network to generate an artificial audio spectrogram that causes the trained discriminator to misclassify the artificial audio spectrogram to be a true audio spectrogram.
4. The method of claim 2 , wherein training the second neural network comprises iteratively training the second neural network to minimize a discriminator adversarial loss that represents the degree to which the discriminator can correctly recognize an input audio spectrogram as artificial, and wherein training the first neural network comprises iteratively training the first neural network to minimize a combination of both a generator adversarial loss that represents the degree to which the discriminator can recognize the artificial audio spectrogram output generated by the generator as artificial, and an identity loss that represents a self-to-self conversion to self.
5. The method of claim 1 , wherein inputting the original audio recording spectrogram comprises processing, by a pre-processing module of the GAN based restoration system, an original audio waveform data structure representing the original audio recording comprising audio imperfections due to physical damage to a medium on which the original audio recording is recorded, to generate the original audio spectrogram and the input target damage vector based on the audio imperfections.
6. The method of claim 5 , wherein the input target damage vector specifies one or more damage characteristics that represent an amount of physical damage to a target medium different from the physical damage to the medium on which the original audio recording is recorded.
7. The method of claim 1 , wherein the input target damage vector is generated by a damage encoder of the GAN based restoration system, based on an input date label, wherein the damage encoder is trained to correlate a date label with a damage embedding represented as the input target damage vector.
8. The method of claim 1 , wherein converting the artificial audio recording spectrogram to the artificial audio recording waveform output comprises introducing, by the spectrogram inversion module, a phase into the artificial audio spectrogram to thereby generate the artificial audio waveform data structure.
9. The method of claim 1 , wherein the input target damage vector corresponds to a target damage level of a restored audio recording having a level of damage less than the original audio recording has, and the method further comprises outputting, by the GAN based restoration system, the artificial audio waveform data structure as a restored version of the original audio recording to one of a storage system for later playback, or to an audio output device for output as an audio output.
10. The method of claim 1 , wherein the input target damage vector corresponds to a target damage level of a retro audio recording having a level of damage greater than the original audio recording has, and the method further comprises outputting, by the GAN based restoration system, the artificial audio waveform data structure as a retro version of the original audio recording to one of a storage system for later playback, or to an audio output device for output as an audio output.
11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a generative adversarial network (GAN) based restoration system that operates to:
train a first neural network, of a generator of the GAN based restoration system, to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector, thereby generating a trained generator;
input, to the trained generator, an original audio recording spectrogram corresponding to an original audio recording and an input target damage vector;
process, by the trained generator, the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector; and
convert, by a spectrogram inversion module of the GAN based restoration system, the artificial audio recording spectrogram to an artificial audio recording waveform output.
12. The computer program product of claim 11 , wherein the computer readable program further causes the GAN based restoration system to:
train a second neural network of a discriminator, of the GAN based restoration system, to discriminate between artificial audio spectrograms and true audio spectrograms, and thereby generate a trained discriminator.
13. The computer program product of claim 12 , wherein training the first neural network of the generator comprises training the first neural network to generate an artificial audio spectrogram that causes the trained discriminator to misclassify the artificial audio spectrogram to be a true audio spectrogram.
14. The computer program product of claim 12 , wherein training the second neural network comprises iteratively training the second neural network to minimize a discriminator adversarial loss that represents the degree to which the discriminator can correctly recognize an input audio spectrogram as artificial, and wherein training the first neural network comprises iteratively training the first neural network to minimize a combination of both a generator adversarial loss that represents the degree to which the discriminator can recognize the artificial audio spectrogram output generated by the generator as artificial, and an identity loss that represents a self-to-self conversion to self.
15. The computer program product of claim 11 , wherein inputting the original audio recording spectrogram comprises processing, by a pre-processing module of the GAN based restoration system, an original audio waveform data structure representing the original audio recording comprising audio imperfections due to physical damage to a medium on which the original audio recording is recorded, to generate the original audio spectrogram and the input target damage vector based on the audio imperfections.
16. The computer program product of claim 15 , wherein the input target damage vector specifies one or more damage characteristics that represent an amount of physical damage to a target medium different from the physical damage to the medium on which the original audio recording is recorded.
17. The computer program product of claim 11 , wherein the input target damage vector is generated by a damage encoder of the GAN based restoration system, based on an input date label, wherein the damage encoder is trained to correlate a date label with a damage embedding represented as the input target damage vector.
18. The computer program product of claim 11 , wherein converting the artificial audio recording spectrogram to the artificial audio recording waveform output comprises introducing, by the spectrogram inversion module, a phase into the artificial audio spectrogram to thereby generate the artificial audio waveform data structure.
19. The computer program product of claim 11 , wherein the input target damage vector corresponds to a target damage level of a restored audio recording having a level of damage less than the original audio recording has, and the computer readable program further causes the GAN based restoration system to output the artificial audio waveform data structure as a restored version of the original audio recording to one of a storage system for later playback, or to an audio output device for output as an audio output.
20. An apparatus comprising:
a processor; and
a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, causes the processor to implement a generative adversarial network (GAN) based restoration system that operates to:
train a first neural network, of a generator of the GAN based restoration system, to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector, thereby generating a trained generator;
input, to the trained generator, an original audio recording spectrogram corresponding to an original audio recording and an input target damage vector;
process, by the trained generator, the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector; and
convert, by a spectrogram inversion module of the GAN based restoration system, the artificial audio recording spectrogram to an artificial audio recording waveform output.