Systems and methods for audio source separation via multi-scale feature learning
Various embodiments of a system and methods for audio source separation via multi-scale feature learning are disclosed.
1. A system, comprising:
a computer-implemented neural-network based architecture, including:
a downstream path configured to receive an input mixture, the downstream path comprising:
a plurality of downstream convolutional blocks configured to learn a plurality of features of the input mixture, wherein each downstream convolutional block of the plurality of downstream convolutional blocks includes a plurality of downstream convolutional layers having exponentially varying dilation rates associated with each respective upstream convolutional layer of the plurality of downstream convolutional layers;
wherein a first convolutional layer of the first downstream convolutional block directly receives the input mixture; and
an upstream path in communication with the downstream path, the upstream path configured to output a plurality of source waveforms associated with the input mixture, the upstream path comprising:
a plurality of upstream convolutional blocks configured to learn a plurality of features of the input mixture, wherein each upstream convolutional block of the plurality of upstream convolutional blocks includes a plurality of upstream convolutional layers having exponentially varying dilation rates associated with each respective upstream convolutional layer of the plurality of upstream convolutional layers;
the input mixture being connected directly to a final convolutional layer by a skip connection;
wherein the plurality of upstream convolutional blocks are transposed relative to the plurality of downstream convolutional blocks.
2. The system of claim 1 , wherein each downstream convolutional layer of the plurality of downstream convolutional layers and each upstream convolutional layer of the plurality of upstream convolutional layers is associated with a respective activation function, wherein each respective activation function is a leaky rectified linear unit (leaky ReLu).
3. The system of claim 1 , wherein each downstream convolutional block of the plurality of downstream convolutional blocks is associated with an index value such that a first downstream convolutional block in a sequence of downstream convolutional blocks is associated with a lowest index value and a last downstream convolutional block in the sequence of downstream convolutional blocks is associated with a highest index value.
4. The system of claim 1 , wherein each upstream convolutional block of the plurality of upstream convolutional blocks is associated with an index value such that a first upstream convolutional block in a sequence of upstream convolutional blocks is associated with a highest index value and a last upstream convolutional block in the sequence of upstream convolutional blocks is associated with a lowest index value.
5. The system of claim 1 , wherein each upstream convolutional layer of the plurality of upstream convolutional layers and each downstream convolutional layer of the plurality of downstream convolutional layers includes a respective plurality of convolution filters, wherein a quantity of convolution filters of the plurality of convolution filters is associated with each upstream convolutional layer of the plurality of upstream convolutional layers and each downstream convolutional layer of the plurality of downstream convolutional layers is dependent upon an index value associated with a corresponding upstream convolutional block or downstream convolutional block.
6. The system of claim 5 , wherein the dilation rate is dependent upon the index value.
7. The system of claim 1 , wherein each downstream convolutional layer of the plurality of downstream convolutional layers and each upstream convolutional layer of the plurality of upstream convolutional layers includes dense connections between each respective downstream convolutional layer of the plurality of downstream convolutional layers and between each respective upstream convolutional layer of the plurality of upstream convolutional layers.
8. The system of claim 1 , wherein each downstream convolutional block of the plurality of downstream convolutional blocks is associated with a corresponding upstream convolutional block of the plurality of upstream convolutional blocks by a respective concatenated skip connection.
9. The system of claim 1 , further comprising:
a bottleneck path in communication with the downstream path and the upstream path, wherein the bottleneck path comprises:
a plurality of one-dimensional convolution layers, wherein each one-dimensional convolution layer of the plurality of one-dimensional convolution layers includes a plurality of filters and a dilation rate of 1.
10. The system of claim 1 , wherein the upstream path further comprises:
a source estimation layer in communication with the plurality of upstream convolutional blocks, the source estimation layer comprising a plurality of one-dimensional convolutional neural network layers.
11. The system of claim 10 , wherein the source estimation layer executes a hyperbolic tangent activation function.
12. The system of claim 1 , wherein each downstream convolutional block of the plurality of downstream convolutional blocks and each upstream convolutional block of the plurality of upstream convolutional blocks is associated with a respective transition block.
13. The system of claim 1 , the computer-implemented neural-network based architecture being operable for estimating K-1 source waveforms of the plurality of source waveforms and approximating a Kth source waveform as a difference between the input mixture and a summation of the K-1 source waveforms.
14. A method, comprising:
providing a computer-implemented neural-network based architecture, including:
a downstream path configured to receive an input mixture, the downstream path comprising:
a plurality of downstream convolutional blocks configured to learn a plurality of features of the input mixture, wherein each downstream convolutional block of the plurality of downstream convolutional blocks includes a plurality of downstream convolutional layers having exponentially varying dilation rates associated with each respective upstream convolutional layer of the plurality of downstream convolutional layers;
wherein a first convolutional layer of the first downstream convolutional block directly receives the input mixture; and
an upstream path in communication with the downstream path, the upstream path configured to output a plurality of source waveforms associated with the input mixture, the upstream path comprising:
a plurality of upstream convolutional blocks configured to learn a plurality of features of the input mixture, wherein each upstream convolutional block of the plurality of upstream convolutional blocks includes a plurality of upstream convolutional layers having exponentially varying dilation rates associated with each respective upstream convolutional layer of the plurality of upstream convolutional layers;
the input mixture being connected directly to a final convolutional layer by a skip connection;
wherein the plurality of upstream convolutional blocks are transposed relative to the plurality of downstream convolutional blocks; and
providing the computer-implemented neural network based architecture with the input mixture, wherein the input mixture comprises a plurality of audio sources;
inferring a plurality of multi-scale audio features from the input mixture using the computer-implemented neural network based architecture; and
predicting constituent audio sources based on the plurality of inferred multi-scale audio features using the computer-implemented neural network based architecture.
15. The method of claim 14 , further comprising:
training the computer-implemented neural network based architecture in a supervised fashion.
16. The method of claim 15 , wherein training of the computer-implemented neural network based architecture is improved by employing a plurality of dense connections between each downstream convolutional layer of the plurality of downstream convolutional layers and each upstream convolutional layer of the plurality of upstream convolutional layers such that a plurality of feature maps produced by each downstream convolutional layer and each upstream convolutional layer are concatenated to subsequent downstream convolutional layers and upstream convolutional layers.
17. The method of claim 16 , wherein training of the computer-implemented neural network based architecture is improved by employing a plurality of concatenated skip connections between corresponding downstream convolutional blocks and corresponding upstream convolutional blocks of the plurality of downstream convolutional blocks and the plurality of upstream convolutional blocks such that an efficiency of extraction of the plurality of multi- scale audio features from the input mixture is improved.
18. The method of claim 14 , the computer-implemented neural-network based architecture being operable for estimating K-1 source waveforms of the plurality of source waveforms and approximating a Kth source waveform as a difference between the input mixture and a summation of the K-1 source waveforms.