Artificial intelligence based audio coding
Techniques are described for coding audio signals. For example, using a neural network, a residual signal is generated for a sample of an audio signal based on inputs to the neural network. The residual signal is configured to excite a long-term prediction filter and/or a short-term prediction filter. Using the long-term prediction filter and/or the short-term prediction filter, a sample of a reconstructed audio signal is determined. The sample of the reconstructed audio signal is determined based on the residual signal generated using the neural network for the sample of the audio signal.
1. An apparatus for reconstructing one or more audio signals, comprising:
a memory configured to store audio data; and
one or more processors implemented in circuitry and coupled to the memory, the one or more processors configured to:
process one or more inputs using a neural network of an audio decoder, the one or more inputs including a long-term prediction for at least one sample of an audio signal, a linear prediction for the at least one sample of the audio signal, and at least one previous sample of a reconstructed audio signal;
generate, based on processing the one or more inputs using the neural network of the audio decoder, a residual signal for the at least one sample of the audio signal, the residual signal being configured to excite at least one of a long-term prediction filter and a short-term prediction filter of the audio decoder; and
determine, using at least one of the long-term prediction filter and the short-term prediction filter, at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated using the neural network for the at least one sample of the audio signal.
2. The apparatus of claim 1 , wherein the audio signal includes a speech signal, and wherein the reconstructed audio signal includes a reconstructed speech signal.
3. The apparatus of claim 1 , wherein the at least one sample of the audio signal includes a single sample.
4. The apparatus of claim 1 , wherein the at least one sample of the audio signal includes a plurality of samples.
5. The apparatus of claim 4 , wherein the plurality of samples are included in a frame of the audio signal.
6. The apparatus of claim 1 , wherein the short-term prediction filter is a short-term linear prediction (LP) filter.
7. The apparatus of claim 6 , wherein the residual signal includes a linear prediction (LP) residual configured to excite the short-term LP filter.
8. The apparatus of claim 7 , wherein the one or more processors are configured to:
determine, using the short-term LP filter, a long-term linear prediction (LTP) residual for the at least one sample of the audio signal, the LTP residual for the at least one sample being determined based on the linear prediction for the at least one sample of the audio signal and the LP residual generated using the neural network.
9. The apparatus of claim 8 , wherein the long-term prediction filter is a long-term linear prediction (LTP) filter.
10. The apparatus of claim 9 , wherein the long-term prediction for the at least one sample of the audio signal includes an LTP prediction, and wherein the one or more processors are configured to:
determine, using the LTP filter, the at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the LTP residual for the at least one sample of the audio signal and the LTP prediction of the at least one sample of the audio signal, wherein the LTP prediction for the at least one sample is determined using a gain and the at least one previous sample of the reconstructed audio signal.
11. The apparatus of claim 1 , wherein the long-term prediction filter is a long-term linear prediction (LTP) filter.
12. The apparatus of claim 11 , wherein the residual signal includes a long-term linear prediction (LTP) residual configured to excite the LTP filter.
13. The apparatus of claim 12 , wherein the long-term prediction for the at least one sample of the audio signal includes an LTP prediction, and wherein the one or more processors are configured to:
determine, using the LTP filter, a linear prediction (LP) residual for the at least one sample of the audio signal, the LP residual for the at least one sample of the audio signal being determined based on the LTP residual generated using the neural network for the at least one sample of the audio signal and the LTP prediction for the at least one sample of the audio signal, wherein the LTP prediction for the at least one sample is determined using a gain and an LP residual of the at least one previous sample of the audio signal.
14. The apparatus of claim 13 , wherein the gain is applied to the LP residual of the at least one previous sample of the audio signal.
15. The apparatus of claim 13 , wherein the short-term prediction filter is a short-term linear prediction (LP) filter.
16. The apparatus of claim 15 , wherein the one or more processors are configured to:
determine, using the short-term LP filter, the at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the linear prediction and the LP residual determined for the at least one sample of the audio signal.
17. The apparatus of claim 16 , wherein the one or more inputs to the neural network further include a previous LTP residual determined by the neural network for the at least one previous sample of the audio signal.
18. The apparatus of claim 16 , wherein the LTP residual generated using the neural network for the at least one sample of the audio signal is combined with the LTP prediction for the at least one sample of the audio signal to generate the LP residual for the at least one sample of the audio signal.
19. The apparatus of claim 16 , wherein the LP residual determined for the at least one sample of the audio signal is combined with the linear prediction for the at least one sample of the audio signal to determine the at least one sample of the reconstructed audio signal.
20. A method of reconstructing one or more audio signals, the method comprising:
processing one or more inputs using a neural network of an audio decoder, the one or more inputs including a long-term prediction for at least one sample of an audio signal, a linear prediction for the at least one sample of the audio signal, and at least one previous sample of a reconstructed audio signal;
generating, based on processing the one or more inputs using the neural network of the audio decoder, a residual signal for the at least one sample of the audio signal, the residual signal being configured to excite at least one of a long-term prediction filter and a short-term prediction filter of the audio decoder; and
determining, using at least one of the long-term prediction filter and the short-term prediction filter, at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated using the neural network for the at least one sample of the audio signal.
21. The method of claim 20 , wherein the at least one sample of the audio signal includes a single sample.
22. The method of claim 20 , wherein the audio signal includes a speech signal, and wherein the reconstructed audio signal includes a reconstructed speech signal.
23. The method of claim 20 , wherein the at least one sample of the audio signal includes a plurality of samples.
24. The method of claim 23 , wherein the plurality of samples are included in a frame of the audio signal.
25. The method of claim 20 , wherein the short-term prediction filter is a short-term linear prediction (LP) filter.
26. The method of claim 25 , wherein the residual signal includes a linear prediction (LP) residual configured to excite the short-term LP filter.
27. The method of claim 26 , further comprising:
determining, using the short-term LP filter, a long-term linear prediction (LTP) residual for the at least one sample of the audio signal, the LTP residual for the at least one sample being determined based on the linear prediction for the at least one sample of the audio signal and the LP residual generated using the neural network.
28. The method of claim 27 , wherein the long-term prediction filter is a long-term linear prediction (LTP) filter.
29. The method of claim 28 , wherein the long-term prediction for the at least one sample of the audio signal includes an LTP prediction, the method further comprising:
determining, using the LTP filter, the at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the LTP residual for the at least one sample of the audio signal and the LTP prediction of the at least one sample of the audio signal, wherein the LTP prediction for the at least one sample is determined using a gain and the at least one previous sample of the reconstructed audio signal.
30. The method of claim 20 , wherein the long-term prediction filter is a long-term linear prediction (LTP) filter.
31. The method of claim 30 , wherein the residual signal includes a long-term linear prediction (LTP) residual configured to excite the LTP filter.
32. The method of claim 31 , wherein the long-term prediction for the at least one sample of the audio signal includes an LTP prediction, the method further comprising:
determining, using the LTP filter, a linear prediction (LP) residual for the at least one sample of the audio signal, the LP residual for the at least one sample of the audio signal being determined based on the LTP residual generated using the neural network for the at least one sample of the audio signal and an LTP prediction for the at least one sample of the audio signal, wherein the LTP prediction for the at least one sample is determined using a gain and an LP residual of the at least one previous sample of the audio signal.
33. The method of claim 32 , wherein the gain is applied to the LP residual of the at least one previous sample of the audio signal.
34. The method of claim 32 , wherein the short-term prediction filter is a short-term linear prediction (LP) filter.
35. The method of claim 34 , further comprising:
determining, using the short-term LP filter, the at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the linear prediction and the LP residual determined for the at least one sample of the audio signal.
36. The method of claim 35 , wherein the one or more inputs to the neural network further include a previous LTP residual determined by the neural network for the at least one previous sample of the audio signal.
37. The method of claim 35 , wherein the LTP residual generated using the neural network for the at least one sample of the audio signal is combined with the LTP prediction for the at least one sample of the audio signal to generate the LP residual for the at least one sample of the audio signal.
38. The method of claim 35 , wherein the LP residual determined for the at least one sample of the audio signal is combined with the linear prediction for the at least one sample of the audio signal to determine the at least one sample of the reconstructed audio signal.
39. A non-transitory computer-readable storage medium storing instructions that when executed cause one or more processors of a device to:
process one or more inputs using a neural network of an audio decoder, the one or more inputs including a long-term prediction for at least one sample of an audio signal, a linear prediction for the at least one sample of the audio signal, and at least one previous sample of a reconstructed audio signal;
generate, based on processing the one or more inputs using the neural network of the audio decoder, a residual signal for the at least one sample of the audio signal, the residual signal being configured to excite at least one of a long-term prediction filter and a short-term prediction filter of the audio decoder; and
determine, using at least one of the long-term prediction filter and the short-term prediction filter, at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated using the neural network for the at least one sample of the audio signal.