Audio coding using machine learning based linear filters and non-linear neural sources
Systems and techniques are described for coding audio signals. For example, a voice decoder can generate, using a first neural network, an excitation signal for at least one sample of an audio signal at least in part by performing a non-linear operation based on one or more inputs to the first neural network, the excitation signal being configured to excite a learned linear filter. The voice decoder can further generate, using the learned linear filter and the excitation signal, at least one sample of a reconstructed audio signal. For example, a second neural network can be used to generate coefficients for one or more learned linear filters, which receive as input the excitation signal generated by the first neural network trained to perform the non-linear operation.
1 . An apparatus for reconstructing one or more audio signals, comprising:
at least one memory configured to store audio data; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
generate, using a first neural network, an excitation signal for at least one sample of an audio signal at least in part by performing a non-linear operation based on one or more inputs to the first neural network, the excitation signal being configured to excite a learned linear filter;
generate, using a second neural network, a parameter for the learned linear filter based on the at least one sample of the audio signal, the parameter specifying characteristics for application by the learned linear filter; and
generate, using the parameter for the learned linear filter and the excitation signal, at least one sample of a reconstructed audio signal.
2 . The apparatus of claim 1 , wherein the one or more inputs to the first neural network include features associated with the audio signal.
3 . The apparatus of claim 2 , wherein the features include log-mel-frequency spectrum features.
4 . The apparatus of claim 1 , wherein the non-linear operation performed using the first neural network is a non-linear transform.
5 . The apparatus of claim 4 , wherein the first neural network is configured to perform the non-linear transform on the one or more inputs to the first neural network and to generate the excitation signal, wherein the excitation signal is generated in a time domain.
6 . The apparatus of claim 1 , wherein the non-linear operation performed using the first neural network is based on a non-linear likelihood speech model.
7 . The apparatus of claim 6 , wherein, to generate the excitation signal using the first neural network, the at least one processor is configured to:
generate, using the one or more inputs to the first neural network, a probability distribution by providing the one or more inputs to the non-linear likelihood speech model;
determine one or more samples from the generated probability distribution; and
generate, using the one or more samples from the generated probability distribution, the excitation signal.
8 . The apparatus of claim 7 , wherein the at least one processor is further configured to modify the excitation signal by modifying a sampling process used to determine the one or more samples from the generated probability distribution.
9 . The apparatus of claim 1 , wherein the learned linear filter comprises a time-varying linear filter, and wherein, to generate the reconstructed audio signal using the learned linear filter, the at least one processor is configured to:
parameterize the learned linear filter with the parameter; and
generate, using the parameterized learned linear filter and the excitation signal, the reconstructed audio signal.
10 . The apparatus of claim 9 , wherein the parameter for the time-varying linear filter includes one or more of an impulse response, a frequency response, or one or more rational transfer function coefficients.
11 . A method of reconstructing one or more audio signals, the method comprising:
generating, using a first neural network, an excitation signal for at least one sample of an audio signal at least in part by performing a non-linear operation based on one or more inputs to the first neural network, the excitation signal being configured to excite a learned linear filter;
generating, using a second neural network, a parameter for the learned linear filter based on the at least one sample of the audio signal, the parameter specifying characteristics for application by the learned linear filter; and
generating, using the parameter for the learned linear filter and the excitation signal, at least one sample of a reconstructed audio signal.
12 . The method of claim 11 , wherein the one or more inputs to the first neural network include features associated with the audio signal.
13 . The method of claim 12 , wherein the features include log-mel-frequency spectrum features.
14 . The method of claim 11 , wherein the non-linear operation performed using the first neural network is a non-linear transform.
15 . The method of claim 14 , wherein the first neural network performs the non-linear transform on the one or more inputs to the first neural network and generates the excitation signal, the excitation signal generated in a time domain.
16 . The method of claim 11 , wherein the non-linear operation performed using the first neural network is based on a non-linear likelihood speech model.
17 . The method of claim 16 , wherein generating the excitation signal using the first neural network comprises:
generating, using the one or more inputs to the first neural network, a probability distribution by providing the one or more inputs to the non-linear likelihood speech model;
determining one or more samples from the generated probability distribution; and
generating, using the one or more samples from the generated probability distribution, the excitation signal.
18 . The method of claim 17 , further comprising modifying the excitation signal by modifying a sampling process used to determine the one or more samples from the generated probability distribution.
19 . The method of claim 11 , wherein the learned linear filter comprises a time-varying linear filter, and wherein generating the reconstructed audio signal using the learned linear filter comprises:
parameterizing the learned linear filter with the parameter; and
generating, using the parameterized learned linear filter and the excitation signal, the reconstructed audio signal.
20 . The method of claim 19 , wherein the parameter for the time-varying linear filter includes one or more of an impulse response, a frequency response, or one or more rational transfer function coefficients.
21 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
generate, using a first neural network, an excitation signal for at least one sample of an audio signal at least in part by performing a non-linear operation based on one or more inputs to the first neural network, the excitation signal being configured to excite a learned linear filter;
generate, using a second neural network, a parameter for the learned linear filter based on the at least one sample of the audio signal, the parameter specifying characteristics for application by the learned linear filter; and
generate, using the parameter for the learned linear filter and the excitation signal, at least one sample of a reconstructed audio signal.
22 . The non-transitory computer-readable storage medium of claim 21 , wherein the one or more inputs to the first neural network include features associated with the audio signal.
23 . The non-transitory computer-readable storage medium of claim 22 , wherein the features include log-mel-frequency spectrum features.
24 . The non-transitory computer-readable storage medium of claim 21 , wherein the non-linear operation performed using the first neural network is a non-linear transform.
25 . The non-transitory computer-readable storage medium of claim 24 , wherein the first neural network performs the non-linear transform on the one or more inputs to the first neural network and generates the excitation signal, the excitation signal generated in a time domain.
26 . The non-transitory computer-readable storage medium of claim 21 , wherein the non-linear operation performed using the first neural network is based on a non-linear likelihood speech model.
27 . The non-transitory computer-readable storage medium of claim 26 , wherein, to generate the excitation signal using the first neural network, the instructions, when executed by the one or more processors, cause the one or more processors to:
generate, using the one or more inputs to the first neural network, a probability distribution by providing the one or more inputs to the non-linear likelihood speech model;
determine one or more samples from the generated probability distribution; and
generate, using the one or more samples from the generated probability distribution, the excitation signal.
28 . The non-transitory computer-readable storage medium of claim 27 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to modify the excitation signal by modifying a sampling process used to determine the one or more samples from the generated probability distribution.
29 . The non-transitory computer-readable storage medium of claim 21 , wherein the learned linear filter comprises a time-varying linear filter, and wherein, to generate the reconstructed audio signal using the learned linear filter, the instructions, when executed by the one or more processors, cause the one or more processors to:
parameterize the learned linear filter with the parameter; and
generate, using the parameterized learned linear filter and the excitation signal, the reconstructed audio signal.
30 . The non-transitory computer-readable storage medium of claim 29 , wherein the parameter for the time-varying linear filter includes one or more of an impulse response, a frequency response, or one or more rational transfer function coefficients.