Systems and methods for unsupervised audio source separation using generative priors
Various embodiments of a system and associated method for audio source separation based on generative priors trained on individual sources. Through the use of projected gradient descent optimization, the present approach simultaneously searches in the source-specific latent spaces to effectively recover the constituent sources. Though the generative priors can be defined in the time domain directly, it was found that using spectral domain loss functions leads to good-quality source estimates.
1. A system for audio source separation, the system comprising:
a processor in communication with a memory, the memory including instructions which, when executed, cause the processor to:
synthesize a reconstructed audio mixture through additive mixing of a plurality of source-specific audio samples generated by a plurality of source-specific data priors based on a plurality of source-specific latent features of a plurality of audio sources of an original audio mixture;
iteratively update the plurality of source-specific latent features through optimization of a spectral-domain loss function between a spectrogram of the reconstructed audio mixture and a spectrogram of the original audio mixture; and
obtain a final estimation vector of each audio source of the original audio mixture based on each source-specific data prior and the updated plurality of source-specific latent features.
2. The system of claim 1 , wherein the memory includes instructions which, when executed, further cause the processor to:
generate, by a source-specific data prior generator, a source-specific data prior for each respective audio source of a plurality of audio sources of an original audio mixture based on a plurality of source-specific latent features of the original audio mixture.
3. The system of claim 2 , wherein the source-specific data prior generator is a generative adversarial network configured to generate a source-specific audio sample based on the source-specific latent features of the original audio mixture.
4. The system of claim 3 , wherein the memory includes instructions which, when executed, further cause the processor to:
sample an audio sample from each respective source-specific data prior of the plurality of source-specific data priors.
5. The system of claim 1 , wherein the memory includes instructions which, when executed, further cause the processor to:
generate the reconstructed audio mixture by additive mixing of each of the plurality of sampled source-specific audio samples obtained using each respective source-specific data prior of the plurality of source-specific data priors.
6. The system of claim 1 , wherein the memory includes instructions which, when executed, further cause the processor to:
apply projected gradient descent to the spectral domain loss function that uses the spectrogram of the reconstructed audio mixture and the spectrogram of the original audio mixture to update the plurality of source-specific latent features.
7. The system of claim 6 , wherein the memory includes instructions which, when executed, further cause the processor to:
minimize a multiresolution spectral loss between log magnitudes of the spectrogram of the reconstructed audio mixture and the spectrogram of the original audio mixture at varying spatial resolutions between the original audio mixture and the reconstructed audio mixture;
minimize an aggregated gradient similarity loss between each respective spectrogram of the reconstructed audio mixture and the original audio mixture to enforce systematic differences between each audio source of the plurality of audio sources within the reconstructed audio mixture and the original audio mixture;
minimize a coherence loss between reconstructed audio mixture is coherent with respect to the original audio mixture; and
minimize a frequency consistency loss between a magnitude spectrogram of the original audio mixture and a magnitude spectrogram of the reconstructed audio mixture.
8. The system of claim 1 , wherein the memory includes instructions which, when executed, further cause the processor to:
obtain a mixture spectrogram representative of a spectral domain of the reconstructed audio mixture and a mixture spectrogram representative of a spectral domain of the original audio mixture.
9. The system of claim 1 , wherein the memory includes instructions which, when executed, further cause the processor to:
constrain each source-specific latent feature to a respective latent feature manifold with each update.
10. The system of claim 1 , wherein the memory includes instructions which, when executed, further cause the processor to:
apply a regularizer to an output of each source-specific data prior for each respective audio source of a plurality of audio sources.
11. A method for audio source separation, the method comprising:
synthesizing, by a processor, a reconstructed audio mixture through additive mixing of a plurality of audio samples generated by a plurality of source-specific data priors based on a plurality of source-specific latent features of a plurality of audio sources of an original audio mixture;
iteratively updating, by the processor, the plurality of source-specific latent features through optimization of a spectral-domain loss function between a spectrogram of the reconstructed audio mixture and a spectrogram of the original audio mixture; and
obtaining, by the processor, a final estimation of each audio source of the original audio mixture based on each source-specific data prior and the updated plurality of source-specific latent features.
12. The method of claim 11 , further comprising:
generating, by a source-specific data prior generator, a source-specific data prior for each respective audio source of a plurality of audio sources of an original audio mixture based on a plurality of source-specific latent features of the original audio mixture.
13. The method of claim 12 , wherein the source-specific data prior generator is a generative adversarial network configured to generate a source-specific audio sample based on the source-specific latent features of the original audio mixture.
14. The method of claim 13 , further comprising:
sampling a source-specific audio sample from each respective source-specific data prior of the plurality of source-specific data priors.
15. The method of claim 11 , further comprising:
generating the reconstructed audio mixture by additive mixing of each of the plurality of sampled source-specific audio samples obtained using each respective source-specific data prior of the plurality of source-specific data priors.
16. The method of claim 11 , further comprising:
applying projected gradient descent to the spectral domain loss function that uses the spectrogram of the reconstructed audio mixture and the spectrogram of the original audio mixture to update the plurality of source-specific latent features.
17. The method of claim 16 , further comprising:
minimizing a multiresolution spectral loss between log magnitudes of the spectrogram of the reconstructed audio mixture and the spectrogram of the original audio mixture at varying spatial resolutions between the original audio mixture and the reconstructed audio mixture;
minimizing an aggregated gradient similarity loss between each respective spectrogram of the reconstructed audio mixture and the original audio mixture to enforce systematic differences between each audio source of the plurality of audio sources within the reconstructed audio mixture and the original audio mixture;
minimizing a coherence loss between reconstructed audio mixture is coherent with respect to the original audio mixture; and
minimizing a frequency consistency loss between a magnitude spectrogram of the original audio mixture and a magnitude spectrogram of the reconstructed audio mixture.
18. The method of claim 11 , further comprising:
obtain a mixture spectrogram representative of a spectral domain of the reconstructed audio mixture and a mixture spectrogram representative of a spectral domain of the original audio mixture.
19. The method of claim 11 , further comprising:
constraining each source-specific latent feature to a respective latent feature manifold with each update.
20. The method of claim 11 , further comprising:
applying a regularizer to an output of each source-specific data prior for each respective audio source of a plurality of audio sources.