Multi-rate end-to-end neural audio upsampler
The present disclosure describes aspects of an end-to-end neural audio upsampler and bandwidth extender. In some aspects, the end-to-end neural audio upsampler and bandwidth extender is configured to receive an input signal having a first bandwidth and generate, using a first neural network model and in a time domain, a feature vector based on the input signal. The end-to-end neural audio upsampler and bandwidth extender is further configured to generate, using a second neural network model and in the time domain, an output signal based on the feature vector, where the output signal has a second bandwidth that is greater than the first bandwidth.
1 . An electronic device, comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive an input signal having a first bandwidth;
generate, using a first neural network model and in a time domain, a feature vector based on the input signal;
generate, using a second neural network model and in the time domain, an output signal based on the feature vector, wherein the output signal has a second bandwidth that is greater than the first bandwidth,
generate, using a third neural network model and in the time domain, a second feature vector based on a second input signal, wherein the second input signal has a third bandwidth; and
generate, using the second neural network model and in the time domain, a second output signal based on the second feature vector, wherein the second output signal has the second bandwidth that is greater than the first and third bandwidths.
2 . The electronic device of claim 1 , wherein the at least one processor is further configured to sample the input signal at a first sampling rate before generating the feature vector.
3 . The electronic device of claim 1 , wherein the at least one processor is further configured to:
receive the second input signal having the third bandwidth different than the first and second bandwidths.
4 . The electronic device of claim 1 , wherein the at least one processor is further configured to train the first neural network model and the second neural network model using a database of speech data and music data.
5 . The electronic device of claim 1 , wherein the first neural network model and the second neural network model are convolutional neural network models.
6 . The electronic device of claim 1 , wherein the first neural network model comprises four strided convolutional neural network layers for downsampling and six convolutional neural network layers with increasing dilation.
7 . The electronic device of claim 1 , wherein the second neural network model comprises four strided convolutional neural network layers for upsampling and six convolutional neural network layers with increasing dilation.
8 . The electronic device of claim 1 , wherein to generate the output signal, the at least one processor is configured to add frequencies above the first bandwidth to the input signal to generate the output signal with the second bandwidth that is greater than the first bandwidth.
9 . A method, comprising:
receiving, by an encoder, an input signal having a first bandwidth;
generating, using a first neural network model of the encoder and in a time domain, a feature vector based on the input signal;
generating, using a second neural network model of a decoder and in the time domain, an output signal based on the feature vector, wherein the output signal has a second bandwidth that is greater than the first bandwidth,
generating, using a third neural network model of a second encoder and in the time domain, a second feature vector based on a second input signal, wherein the second input signal has a third bandwidth; and
generating, using the second neural network model of the decoder and in the time domain, a second output signal based on the second feature vector, wherein the second output signal has the second bandwidth that is greater than the first and third bandwidths.
10 . The method of claim 9 , further comprising sampling, using the encoder, the input signal at a first sampling rate before generating the feature vector.
11 . The method of claim 9 , further comprising:
receiving, at the second encoder, the second input signal having the third bandwidth different than the first and second bandwidths.
12 . The method of claim 9 , further comprising training the first neural network model and the second neural network model using a database of speech data and music data.
13 . The method of claim 12 , wherein the first neural network model and the second neural network model are convolutional neural network models.
14 . The method of claim 9 , wherein the first neural network model comprises four strided convolutional neural network layers for downsampling and six convolutional neural network layers with increasing dilation.
15 . The method of claim 9 , wherein the second neural network model comprises four strided convolutional neural network layers for upsampling and six convolutional neural network layers with increasing dilation.
16 . The method of claim 9 , wherein generating the output signal comprises adding frequencies above the first bandwidth to the input signal to generate the output signal with the second bandwidth that is greater than the first bandwidth.
17 . The method of claim 9 , wherein the encoder is part of a first electronic device, and the method further comprising:
generating, using a quantizer of the first electronic device, a quantized feature vector based on the feature vector; and
transmitting the quantized feature vector to a second electronic device,
wherein the decoder is part of the second electronic device and is configured to generate the output signal based on the quantized feature vector.
18 . A non-transitory computer-readable medium storing instructions that, when executed by a processor of an electronic device, cause the electronic device to perform operations comprising:
receiving, by an encoder, an input signal having a first bandwidth;
generating, using a first neural network model of the encoder and in a time domain, a feature vector based on the input signal, wherein the first neural network model comprises a plurality of encoder blocks with respective downsampling factors;
generating, using a second neural network model of a decoder and in the time domain, an output signal based on the feature vector, wherein the output signal has a second bandwidth that is greater than the first bandwidth, and wherein the second neural network model comprises a plurality of decoder blocks with respective upsampling factors;
generating, using a third neural network model of a second encoder and in the time domain, a second feature vector based on a second input signal, wherein the second input signal has a third bandwidth; and
generating, using the second neural network model of the decoder and in the time domain, a second output signal based on the second feature vector, wherein the second output signal has the second bandwidth that is greater than the first and third bandwidths.
19 . The non-transitory computer-readable medium of claim 18 , the operations further comprising:
receiving, at the second encoder, the second input signal having the third bandwidth different than the first and second bandwidths,
wherein the third neural network model comprises a second plurality of encoder blocks with respective downsampling factors.
20 . The non-transitory computer-readable medium of claim 18 , wherein:
generating the output signal comprises adding frequencies above the first bandwidth to the input signal to generate the output signal with the second bandwidth that is greater than the first bandwidth,
the first neural network model comprises four strided convolutional neural network layers for downsampling and six convolutional neural network layers with increasing dilation, and
the second neural network model comprises the four strided convolutional neural network layers for upsampling and the six convolutional neural network layers with increasing dilation.