Perceptually-based loss functions for audio encoding and decoding based on machine learning
Computer-implemented methods for training a neural network, as well as for implementing audio encoders and decoders via trained neural networks, are provided. The neural network may receive an input audio signal, generate an encoded audio signal and decode the encoded audio signal. A loss function generating module may receive the decoded audio signal and a ground truth audio signal, and may generate a loss function value corresponding to the decoded audio signal. Generating the loss function value may involve applying a psychoacoustic model. The neural network may be trained based on the loss function value. The training may involve updating at least one weight of the neural network.
1. A method of encoding an audio signal using a neural network, the method comprising:
receiving the audio signal; and
encoding the audio signal via the neural network by using at least one weighted parameter of the neural network, wherein the neural network is trained based on a perceptually-based loss function, the perceptually-based loss function modeling an acoustic response of a human ear canal, and wherein the at least one weighted parameter is determined by the perceptually-based loss function.
2. The method of claim 1 , wherein encoding the audio signal via the neural network comprises:
generating, by the neural network and based on the audio signal, the encoded audio signal.
3. The method of claim 1 , further comprising:
prior to the encoding, transforming the audio signal from a time domain representation to a frequency domain representation.
4. The method of claim 1 , wherein the perceptually-based loss function is based on at least one selected from a group consisting of a psychoacoustic model, a psychoacoustic masking threshold, an average noise-to-masking ratio, minimizing an average noise-to-masking ratio, modeling an outer ear transfer function, grouping into critical bands, frequency-domain masking, level dependent spreading, and modeling of a frequency-dependent hearing threshold.
5. The method of claim 1 , wherein the audio signal comprises at least one selected from a group consisting of speech, music, and a mixture of sounds.
6. The method of claim 1 , wherein the neural network comprises an autoencoder.
7. The method of claim 1 , further comprising:
outputting the encoded audio signal in a compressed audio format.
8. A method of decoding an audio signal using a neural network, the method comprising:
receiving an encoded audio signal; and
decoding the encoded audio signal via the neural network to produce at least one selected from a group consisting of a decoded audio signal and at least one decoded transform coefficient by using at least one weighted parameter of the neural network, wherein the neural network is trained based on a perceptually-based loss function, the perceptually-based loss function modeling an acoustic response of a human ear canal, and wherein the at least one weighted parameter is determined by the perceptually-based loss function.
9. The method of claim 8 , wherein the decoding comprises:
generating, by the neural network and based on the encoded audio signal, the at least one selected from the group consisting of the decoded audio signal and the at least one decoded transform coefficient.
10. The method of claim 8 , wherein the perceptually-based loss function is based on at least one selected from a group consisting of a psychoacoustic model, a psychoacoustic masking threshold, an average noise-to-masking ratio, minimizing an average noise-to-masking ratio, modeling an outer ear transfer function, grouping into critical bands, frequency-domain masking, level dependent spreading, and modeling of a frequency dependent hearing threshold.
11. The method of claim 8 , wherein the decoded audio signal comprises at least one selected from a group consisting of speech, music, and a mixture of sounds.
12. An audio decoder comprising:
a processor; and
non-transitory storage media coupled to the processor, the non-transitory storage media storing instructions executable by the processor to:
receive an encoded audio signal; and
generate, via a neural network, at least one selected from a group consisting of a decoded audio signal and at least one decoded transform coefficient based on the received encoded audio signal by using at least one weighted parameter of the neural network, wherein the neural network is trained based on a perceptually-based loss function, the perceptually-based loss function modeling an acoustic response of a human ear canal, and wherein the at least one weighted parameter is determined by the perceptually-based loss function.
13. The audio decoder of claim 12 , wherein the neural network comprises:
a plurality of neuron layers comprising:
an input neuron layer;
a plurality of hidden neuron layers; and
an output neuron layer.
14. The audio decoder of claim 13 , wherein each neuron layer of the plurality of neuron layers are configured with at least one selected from a group consisting of a rectified linear unit (ReLU) activation function, a sigmoidal activation function, a tanh activation function, and an exponential linear unit (ELU) activation function.
15. The audio decoder of claim 12 , wherein the neural network is configured based on a perceptually-based loss function.
16. The audio decoder of claim 15 , wherein the perceptually-based loss function is based on at least one selected from a group consisting of a psychoacoustic model, a psychoacoustic masking threshold, an average noise-to-masking ratio, minimizing an average noise-to-masking ratio, modeling an outer ear transfer function, grouping into critical bands, frequency-domain masking, level dependent spreading, and modeling of a frequency-dependent hearing threshold.
17. The audio decoder of claim 12 , wherein the decoded audio signal comprises at least one selected from a group consisting of speech, music, and a mixture of sounds.
18. The audio decoder of claim 12 , wherein the encoded audio signal is configured in a compressed audio format.