Method, apparatus and system for neural network hearing aid
The disclosure generally relates to a method, system and apparatus to improve a user's understanding of speech in real-time conversations by processing the audio through a neural network contained in a hearing device. The hearing device may be a headphone or hearing aid. In one embodiment, the disclosure relates to an apparatus to enhance incoming audio signal. The apparatus includes a controller to receive an incoming signal and provide a controller output signal; a neural network engine (NNE) circuitry in communication with the controller, the NNE circuitry activatable by the controller, the NNE circuitry configured to generate an NNE output signal from the controller output signal; and a digital signal processing (DSP) circuitry to receive one or more of controller output signal or the NNE circuitry output signal to thereby generate a processed signal; wherein the controller determines a processing path of the controller output signal through one of the DSP or the NNE circuitries as a function of one or more of predefined parameters, incoming signal characteristics and NNE circuitry feedback.
1. A method of processing an audio signal with an ear-worn device including a microphone, a controller coupled to the microphone, a neural network engine (NNE) coupled to the controller, and an output signal generator coupled to the NNE, the method comprising:
detecting the audio signal with the microphone of the ear-worn device;
providing the audio signal detected by the microphone of the ear-worn device to the controller of the ear-worn device;
processing, with the controller of the ear-worn device, the audio signal to estimate a metric characterizing an aspect of an acoustic environment of the ear-worn device;
determining, with the controller of the ear-worn device, whether the estimated metric satisfies at least one criterion;
in dependence on determining whether the estimated metric satisfies the at least one criterion:
denoising the audio signal at least in part by applying to the audio signal a level of denoising that is less than a maximum level of denoising achievable by the NNE; or
providing the audio signal to the output signal generator without denoising the audio signal with the NNE; and
outputting the audio signal or the denoised audio signal with the output signal generator of the ear-worn device.
2. The method of claim 1 , wherein processing the audio signal to estimate the metric characterizing the aspect of the acoustic environment of the ear-worn device comprises processing the audio signal to estimate a signal-to-noise ratio (SNR) of the audio signal.
3. The method of claim 1 , wherein the level of denoising that is less than the maximum level of denoising achievable by the NNE is a first level of denoising, and wherein the method further comprises:
based on determining that the estimated metric satisfies the at least one criterion, denoising the audio signal at least in part by applying to the audio signal a second level of denoising that is greater than the first level of denoising.
4. The method of claim 1 , wherein denoising the audio signal comprises denoising the audio signal to obtain a denoised audio signal having a signal-to-noise ratio (SNR) that is lower than a target SNR.
5. The method of claim 4 , wherein the method further comprises:
based on determining that the estimated metric satisfies the at least one criterion, denoising the audio signal with the NNE to obtain a denoised audio signal having an SNR that is substantially equivalent to the target SNR.
6. The method of claim 1 , wherein the level of denoising that is less than the maximum level of denoising achievable by the NNE is a first level of denoising, wherein the audio signal comprises multiple components including a noise component, and
wherein denoising the audio signal comprises:
denoising the audio signal to obtain an intermediate audio signal at least in part by applying to the audio signal a second level of denoising that is greater than the first level of denoising; and
combining the intermediate audio signal with at least a portion of the noise component to obtain the denoised audio signal.
7. The method of claim 1 , wherein denoising the audio signal with the NNE comprises:
determining, with the NNE, a maximum achievable signal-to-noise ratio (SNR), wherein the maximum achievable SNR represents a maximum SNR that the NNE can achieve while satisfying one or more performance criteria; and
denoising the audio signal based at least in part on the maximum achievable SNR.
8. The method of claim 1 , wherein denoising the audio signal with the NNE further comprises:
estimating, with the NNE of the ear-worn device, a likelihood that an output of the NNE will satisfy a performance metric; and
denoising the audio signal based at least in part on the estimated likelihood.
9. The method of claim 1 , further comprising:
selectively providing, with the controller of the ear-worn device, the audio signal to a first signal processing path or a second signal processing path, the first signal processing path including a signal processing path through the NNE of the ear-worn device, and the second signal processing path excluding a signal processing path through the NNE of the ear-worn device.
10. The method of claim 9 , wherein selectively providing the audio signal to the first signal processing path or the second signal processing path is based on the estimated metric characterizing the aspect of the acoustic environment.
11. The method of claim 9 , wherein the first signal processing path and the second signal processing path each include a signal processing path through a digital signal processing (DSP) circuit of the ear-worn device.
12. An ear-worn device, comprising:
a microphone configured to detect an audio signal and provide the audio signal to a controller of the ear-worn device;
the controller, wherein the controller is configured to:
process the audio signal to estimate a metric characterizing an aspect of an acoustic environment of the ear-worn device; and
determine whether the estimated metric satisfies at least one criterion;
a neural network engine (NNE) configured to denoise the audio signal, the denoising comprising:
in dependence on determining whether the estimated metric satisfies the at least one criterion, denoising the audio signal at least in part by applying to the audio signal a level of denoising that is less than a maximum level of denoising achievable by the NNE; and
an output signal generator configured to output the denoised audio signal.
13. The ear-worn device of claim 12 , wherein the controller is further configured to:
based on determining that the estimated metric satisfies the at least one criterion, denoising the audio signal at least in part by applying to the audio signal the maximum level of denoising achievable by the NNE.
14. The ear-worn device of claim 12 , wherein the audio signal comprises multiple components including a noise component, wherein the level of denoising that is less than the maximum level of denoising achievable by the NNE is a first level of denoising, and
wherein denoising the audio signal comprises:
denoising the audio signal to obtain an intermediate audio signal at least in part by applying to the audio signal a second level of denoising that is greater than the first level of denoising; and
combining the intermediate audio signal with at least a portion of the noise component to obtain the denoised audio signal.
15. The ear-worn device of claim 12 , wherein the NNE is further configured to:
determine a maximum achievable signal-to-noise ratio (SNR), wherein the maximum achievable SNR represents a maximum SNR that the NNE can achieve while satisfying one or more performance criteria; and
denoise the audio signal based on the maximum achievable SNR.
16. The ear-worn device of claim 12 , wherein the NNE is further configured to:
estimate a likelihood that an output of the NNE will satisfy a performance metric; and
denoise the audio signal based at least in part on the estimated likelihood.
17. The ear-worn device of claim 12 , further comprising:
a first signal processing path through the NNE of the ear-worn device; and
a second signal processing path that excludes a signal processing path through the NNE, wherein the controller is further configured to selectively provide the audio signal to the first signal processing path or the second signal processing path.
18. The ear-worn device of claim 17 , wherein the first signal processing path and the second signal processing path each include a signal processing path through a digital signal processing (DSP) circuit of the ear-worn device.
19. The ear-worn device of claim 17 , wherein the controller is configured to selectively provide the audio signal to the first signal processing path or the second signal processing path based on the metric characterizing the aspect of the acoustic environment.
20. At least one non-transitory computer-readable medium storing instructions that, when executed by computing hardware, cause the computing hardware to perform a method of processing an audio signal with an ear-worn device including a microphone, a controller coupled to the microphone, a neural network engine (NNE) coupled to the controller, and an output signal generator coupled to the NNE, the method comprising:
detecting the audio signal with the microphone of the ear-worn device;
providing the audio signal detected by the microphone of the ear-worn device to the controller of the ear-worn device;
processing the audio signal to estimate a metric characterizing an aspect of an acoustic environment of the ear-worn device;
determining whether the estimated metric satisfies at least one criterion;
in dependence on determining whether the estimated metric satisfies the at least one criterion:
denoising the audio signal at least in part by applying to the audio signal a level of denoising that is less than a maximum level of denoising achievable by the NNE; or
providing the audio signal to the output signal generator without denoising the audio signal with the NNE; and
outputting the audio signal or the denoised audio signal with the output signal generator of the ear-worn device.