Method, apparatus and system for neural network hearing aid
The disclosure generally relates to a method, system and apparatus to improve a user's understanding of speech in real-time conversations by processing the audio through a neural network contained in a hearing device. The hearing device may be a headphone or hearing aid. In one embodiment, the disclosure relates to an apparatus to enhance incoming audio signal. The apparatus includes a controller to receive an incoming signal and provide a controller output signal; a neural network engine (NNE) circuitry in communication with the controller, the NNE circuitry activatable by the controller, the NNE circuitry configured to generate an NNE output signal from the controller output signal; and a digital signal processing (DSP) circuitry to receive one or more of controller output signal or the NNE circuitry output signal to thereby generate a processed signal; wherein the controller determines a processing path of the controller output signal through one of the DSP or the NNE circuitries as a function of one or more of predefined parameters, incoming signal characteristics and NNE circuitry feedback.
1. A hearing aid system, comprising:
an ear-worn device including:
a microphone configured to receive an audible signal and output an electrical signal representing the audible signal;
front-end circuitry coupled to the microphone and configured to receive the electrical signal representing the audible signal, digitize the electrical signal, and output a digitized version of the audible signal;
a controller configured to:
receive the digitized version of the audible signal;
receive a user mode selection; and
selectively output the digitized version of the audible signal to either a digital signal processor (DSP) or a neural network engine comprising a neural network depending at least in part on the user mode selection;
the neural network engine, wherein the neural network engine is coupled to an output of the controller and configured to:
determine a target signal-to-noise ratio (SNR) for a combined signal that will result from processing the digitized version of the audible signal with the neural network;
separate the digitized version of the audible signal into multiple source signals;
apply gains to the multiple source signals based at least in part on the target SNR determined for the combined signal;
create the combined signal by recombining the multiple source signals after application of the gains; and
provide the combined signal to the DSP;
the DSP, wherein the DSP is coupled to the output of the controller and an output of the neural network engine, and wherein the DSP is configured to, upon receiving the combined signal from the neural network engine, apply frequency-dependent amplification to the combined signal to generate an output signal; and
a speaker, coupled to an output of the DSP and configured to playback the output signal in audible form.
2. The hearing aid system of claim 1 , wherein the controller is further configured to determine a signal-to-noise ratio (SNR) of the digitized version of the audible signal, and wherein the controller is further configured to selectively output the digitized version of the audible signal to either the DSP or the neural network engine depending at least in part on the SNR of the digitized version of the audible signal.
3. The hearing aid system of claim 2 , wherein the target SNR for the combined signal is determined based at least in part on the SNR of the digitized version of the audible signal.
4. The hearing aid system of claim 2 , wherein the controller is further configured to:
compare the SNR of the digitized version of the audible signal to a threshold; and
selectively output the digitized version of the audible signal to the DSP when the SNR of the digitized version of the audible signal exceeds the threshold.
5. The hearing aid system of claim 1 , wherein the front-end circuitry, controller, neural network engine, and DSP are implemented on a system-on-chip.
6. The hearing aid system of claim 1 , wherein the neural network engine is further configured to compare the target SNR determined for the combined signal to an indication of an amount of denoising, and to select the gains based at least in part on a result of the comparison.
7. The hearing aid system of claim 6 , wherein the neural network engine is configured to, upon determining from the comparison of the target SNR determined for the combined signal to the indication of the amount of denoising that the amount of denoising is unachievable, target a signal-to-noise ratio of the combined signal that is less than that achievable by the neural network engine.
8. The hearing aid system of claim 1 , wherein the neural network engine is further configured to receive an indication of a user-selected directionality and to select the gains based at least in part on the user-selected directionality.
9. The hearing aid system of claim 1 , wherein the DSP is configured to apply frequency-dependent amplification including the application of gains to different frequency bands of the combined signal.
10. The hearing aid system of claim 1 , wherein the neural network is a recurrent neural network.
11. The hearing aid system of claim 1 , wherein the neural network engine is configured to provide a feedback signal to the controller, and wherein the controller is further configured to selectively output the digitized version of the audible signal to either the DSP or the neural network engine depending at least in part on the feedback signal received from the neural network engine.
12. The hearing aid system of claim 11 , wherein the feedback signal represents at least a portion of the combined signal.
13. The hearing aid system of claim 1 , wherein the controller is configured to provide the digitized version of the audible signal to the neural network engine in segments, and wherein the neural network engine is configured to process a segment of the digitized version of the audible signal in a time less than or equal to a duration of the segment.
14. The hearing aid system of claim 1 , further comprising a housing configured to house the microphone, front-end circuitry, controller, neural network engine, DSP, and speaker.
15. The hearing aid system of claim 14 , further comprising a power source disposed within the housing and coupled to the controller.
16. The hearing aid system of claim 15 , wherein the neural network engine is further configured to select the gains based at least on a status of the power source.
17. The hearing aid system of claim 1 , further comprising a mobile computing device configured to communicate wirelessly with the ear-worn device, wherein the mobile computing device is configured to provide to the ear-worn device an indication of an amount of denoising.
18. The hearing aid system of claim 17 , wherein the mobile computing device is further configured to provide to the ear-worn device the user mode selection.
19. The hearing aid system of claim 1 , wherein the neural network is a convolutional neural network.
20. The hearing aid system of claim 19 , wherein the neural network comprises a gated recurrent unit (GRU) layer.