Method, apparatus and system for neural network hearing aid
The disclosure generally relates to a method, system and apparatus to improve a user's understanding of speech in real-time conversations by processing the audio through a neural network contained in a hearing device. The hearing device may be a headphone or hearing aid. In one embodiment, the disclosure relates to an apparatus to enhance incoming audio signal. The apparatus includes a controller to receive an incoming signal and provide a controller output signal; a neural network engine (NNE) circuitry in communication with the controller, the NNE circuitry activatable by the controller, the NNE circuitry configured to generate an NNE output signal from the controller output signal; and a digital signal processing (DSP) circuitry to receive one or more of controller output signal or the NNE circuitry output signal to thereby generate a processed signal; wherein the controller determines a processing path of the controller output signal through one of the DSP or the NNE circuitries as a function of one or more of predefined parameters, incoming signal characteristics and NNE circuitry feedback.
1. A hearing system configured to enhance incoming audio signals, the hearing system comprising:
an integrated circuit (IC) having formed thereon:
a frontend receiver configured to:
receive an incoming audio signal; and
output a controller input signal based on the incoming audio signal;
a controller coupled to the frontend receiver configured to:
receive the controller input signal;
generate a controller output signal;
determine a signal processing path of the controller output signal based on neural network engine (NNE) circuit feedback, the signal processing path comprising a first signal processing path and/or a second signal processing path of the hearing system; and
selectively transmit the controller output signal to the first signal processing path and/or the second signal processing path of the hearing system;
an NNE circuit defining a first portion of the first signal processing path and configured to generate an NNE output signal based on the controller output signal; and
a digital signal processing (DSP) circuit defining:
a second portion of the first signal processing path, and
a portion of the second signal processing path, wherein the DSP circuit is configured to process the controller output signal and/or the NNE output signal to thereby generate a processed signal.
2. The hearing system of claim 1 , wherein the incoming audio signal has a plurality of signal components, wherein a signal component of the plurality of signal components corresponds to a respective signal source, and
wherein the frontend receiver is further configured to:
process the incoming audio signal to generate the controller input signal; and
transmit, to the controller, the controller input signal, the plurality of signal components including a speech component and/or a noise component.
3. The hearing system of claim 1 , wherein the incoming audio signal has a plurality of signal components, wherein a signal component of the plurality of signal components corresponds to a respective signal source, wherein the NNE circuit is configured to adjust relative volumes of the plurality of signal components, and wherein the DSP circuit is configured to apply a time-varying gain to the controller output signal and/or the NNE output signal.
4. The hearing system of claim 3 , wherein the DSP circuit is configured to apply a different gain to each frequency band of a plurality of frequency bands of the controller output signal and/or the NNE output signal.
5. The hearing system of claim 3 , wherein the plurality of signal components comprises a speech component and a noise component, and wherein the NNE circuit is configured to increase a volume of the speech component relative to a volume of the noise component.
6. The hearing system of claim 1 , further comprising a backend receiver formed on the IC, wherein the backend receiver is configured to:
receive the processed signal from the DSP circuit; and
output, based on the processed signal, an audible signal.
7. The hearing system of claim 6 , wherein:
the hearing system defines a hearing aid, a headphone, or face-worn glasses, and
the backend receiver is configured to output the audible signal less than 15 milliseconds after the frontend receiver receives the incoming audio signal.
8. The hearing system of claim 1 , wherein the IC comprises a System-on-Chip (SOC).
9. The hearing system of claim 8 , further comprising a housing configured to house the SOC and a power source.
10. The hearing system of claim 1 , wherein the controller is configured to determine a signal processing path based on specified criteria, an incoming audio signal characteristic, and/or NNE circuit feedback, wherein the signal processing path comprises the first signal processing path and/or the second signal processing path of the hearing system.
11. The hearing system of claim 10 , further comprising a wireless communication system configured to receive the specified criteria.
12. The hearing system of claim 1 , wherein the NNE circuit is further configured to selectively apply a ratio mask to the controller output signal to obtain a plurality of components, wherein a first component of the plurality of components corresponds to a class of sounds.
13. The hearing system of claim 12 , wherein the NNE circuit is further configured to:
adjust, based on a target signal-to-noise ratio (SNR), a volume of a component of the plurality of components relative to at least one other component of the plurality of components; and
combine the plurality of components into the NNE output signal.
14. The hearing system of claim 13 , wherein the plurality of components comprises a speech component and a noise component, and wherein the NNE circuit is configured to increase a volume of the speech component relative to a volume of the noise component.
15. The hearing system of claim 13 , wherein the plurality of components comprises a user speech component and other sound source components, and wherein the NNE circuit is configured to decrease a volume of the user speech component relative to volumes of the other sound source components.
16. The hearing system of claim 13 , wherein the plurality of components comprises different sound source components, and wherein the NNE circuit is further configured to set respective volumes of the different sound source components.
17. The hearing system of claim 16 , wherein the plurality of components comprises a stationary noise component and a non-stationary noise component, and wherein the NNE circuit is further configured to apply a lower gain to the stationary noise component than the non-stationary noise component.
18. The hearing system of claim 1 , wherein the second signal processing path excludes the NNE circuit.
19. The hearing system of claim 1 , wherein the controller comprises:
one or more processor circuits; and
one or more memory circuits storing processor-executable instructions that, when executed by the one or more processor circuits, cause the one or more processor circuits to perform the:
receiving the controller input signal;
generating the controller output signal; and
selectively transmitting the controller output signal to the first signal processing path and/or the second signal processing path of the hearing system.
20. A method for enhancing incoming audio signals, the method comprising:
using an integrated circuit (IC) of a hearing system, the IC having a frontend receiver, a controller coupled to the frontend receiver, a neural network engine (NNE) circuit, and a digital signal processing (DSP) circuit formed thereon, wherein using the IC comprises:
receiving, at the frontend receiver of the IC, an incoming audio signal;
generating, at the frontend receiver, based on the incoming audio signal, a controller input signal;
using the controller of the IC to perform:
receiving the controller input signal;
generating a controller output signal;
determining a signal processing path for the controller output signal as a function of NNE circuit feedback, wherein the signal processing path comprises a first signal processing path and/or a second signal processing path; and
selectively transmitting the controller output signal to the first signal processing path and/or the second signal processing path of the hearing system; using the NNE circuit of the IC to perform:
upon receiving the controller output signal from the controller, generating an NNE output signal based on the controller output signal, the NNE circuit defining a first portion of the first signal processing path; and
using the DSP circuit of the IC to perform:
processing the controller output signal and/or the NNE output signal to thereby generate a processed signal, the DSP circuit defining a second portion of the first signal processing path and a portion of the second signal processing path.
21. The method of claim 20 , wherein the incoming audio signal has a plurality of signal components, a signal component of the plurality of signal components corresponding to a respective signal source, and wherein using the IC further comprises:
processing, at a front end receiver of the IC, the incoming audio signal to generate the controller input signal; and
transmitting, to the controller, the controller input signal, wherein the plurality of signal components include a speech component and/or a noise component.
22. The method of claim 20 , wherein the incoming audio signal has a plurality of signal components, wherein a signal component of the plurality of signal components corresponds to a respective signal source, and wherein using the IC further comprises:
adjusting, using the NNE circuit, relative volumes of the plurality of signal components; and
applying, using the DSP circuit, a time-varying gain to the controller output signal and/or the NNE output signal.
23. The method of claim 22 , wherein using the IC further comprises:
using the DSP circuit to perform:
applying a different gain to each frequency band of a plurality of frequency bands of the controller output signal and/or the NNE output signal.
24. The method of claim 22 , wherein the plurality of signal components comprises a speech component and a noise component, and wherein using the IC further comprises:
using the NNE circuit to perform:
increasing a volume of the speech component relative to a volume of the noise component.
25. The method of claim 20 , wherein the incoming audio signal has a plurality of signal components, a signal component of the plurality of signal components corresponding to a respective signal source, wherein the IC further comprises a backend receiver formed thereon, and wherein using the IC further comprises:
using the backend receiver to perform:
receiving, from the DSP circuit, the processed signal; and
outputting, based on the processed signal, an audible signal.
26. The method of claim 25 , wherein the hearing system defines a hearing aid, a headphone, and/or face-worn glasses, and wherein using the IC further comprises:
using the backend receiver to perform:
outputting the audible signal less than 15 milliseconds after the frontend receiver receives the incoming audio signal.
27. The method of claim 20 , wherein the IC comprises a System-on-Chip (SOC).
28. The method of claim 27 , wherein the hearing system further comprises a housing configured to house the SOC and a power source.
29. The method of claim 20 , wherein using the IC further comprises:
determining, at the controller, a signal processing path of the controller output signal as a function of specified criteria, an incoming audio signal characteristic, and/or NNE circuit feedback, wherein the signal processing path comprises the first signal processing path and/or the second signal processing path of the hearing system.
30. The method of claim 20 , wherein using the IC further comprises:
using the NNE circuit to perform:
selectively applying a ratio mask to the controller output signal to obtain a plurality of components wherein a first component of the plurality of components corresponds to a class of sounds.
31. The method of claim 30 , wherein using the IC further comprises:
using the NNE circuit to perform:
adjusting, based on a target signal-to-noise ratio, a volume of a component of the plurality of components relative to at least one other component of the plurality of components; and
combining the plurality of components into the NNE output signal.
32. The method of claim 31 , wherein the plurality of components comprises a speech component and a noise component, and wherein using the IC further comprises using the NNE circuit to perform:
increasing a volume of the speech component relative to a volume of the noise component.
33. The method of claim 31 , wherein the plurality of components further comprises a user speech component and other sound source components, and wherein using the IC further comprises using the NNE circuit to perform:
decreasing a volume of the user speech component relative to volumes of the other sound source components.
34. The method of claim 31 , wherein the plurality of components comprises different sound source components, and wherein using the IC further comprises using the NNE circuit to perform:
setting respective volumes of the different sound source components.
35. The method of claim 34 , wherein the plurality of components comprises a stationary noise component and a non-stationary noise component, and wherein using the IC further comprises using the NNE circuit to perform:
applying a lower gain to the stationary noise component than the non-stationary noise component.
36. The method of claim 20 , wherein the second signal processing path of the hearing system excludes the NNE circuit.
37. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by an integrated circuit (IC) of a hearing system, cause the IC to perform a method for enhancing incoming audio signals, having a frontend receiver, a controller coupled to the frontend receiver, a neural network engine (NNE) circuit, and a digital signal processing (DSP) circuit formed thereon, the method comprising:
receiving, at the frontend receiver of the IC, an incoming audio signal;
generating, at the frontend receiver, based on the incoming audio signal, a controller input signal;
using the controller of the IC to perform:
receiving the controller input signal;
generating a controller output signal;
determining, at the controller, a signal processing path for the controller output signal as a function of NNE circuit feedback, wherein the signal processing path comprises a first signal processing path and/or a second signal processing path; and
selectively transmitting the controller output signal to the first signal processing path and/or the second signal processing path of the hearing system;
using the NNE circuit of the IC to perform:
upon receiving the controller output signal from the controller, generating an NNE output signal based on the controller output signal, the NNE circuit defining a first portion of the first signal processing path; and
using the DSP circuit of the IC to perform:
processing the controller output signal and/or the NNE output signal to thereby generate a processed signal, the DSP circuit defining a second portion of the first signal processing path and a portion of the second signal processing path.
38. The at least one non-transitory computer-readable storage medium of claim 37 , wherein the incoming audio signal has a plurality of signal components, a signal component of the plurality of signal components corresponding to a respective signal source, and wherein the method further comprises:
using the frontend receiver of the IC to perform:
processing the incoming audio signal to generate the controller input signal; and
transmitting, to the controller, the controller input signal, wherein the plurality of signal components include a speech component and/or a noise component.
39. The at least one non-transitory computer-readable storage medium of claim 37 , wherein the incoming audio signal has a plurality of signal components, wherein a signal component of the plurality of signal components corresponds to a respective signal source, and wherein the method further comprises:
using the NNE circuit to perform:
adjusting relative volumes of the plurality of signal components; and
using the DSP circuit to perform:
applying a time-varying gain to the controller output signal and/or the NNE output signal.
40. The at least one non-transitory computer-readable storage medium of claim 39 , wherein the method further comprises:
using DSP circuit to perform:
applying a different gain to each frequency band of a plurality of frequency bands of the controller output signal and/or the NNE output signal.
41. The at least one non-transitory computer-readable storage medium of claim 39 , wherein the plurality of signal components comprises a speech component and a noise component, and wherein the method further comprises:
using the NNE circuit to perform:
increasing a volume of the speech component relative to a volume of the noise component.
42. The at least one non-transitory computer-readable storage medium of claim 37 , wherein the incoming audio signal has a plurality of signal components, a signal component of the plurality of signal components corresponding to a respective signal source, and wherein the IC further comprises a backend receiver formed thereon, the method further comprising:
using the backend receiver to perform:
receiving, from the DSP circuit, the processed signal; and
outputting, based on the processed signal, an audible signal.
43. The at least one non-transitory computer-readable storage medium of claim 42 , wherein the hearing system defines a hearing aid, a headphone, and/or face-worn glasses, and wherein the method further comprises:
using the backend receiver to perform:
outputting the audible signal less than 15 milliseconds after the frontend receiver receives the incoming audio signal.
44. The at least one non-transitory computer-readable storage medium of claim 37 , wherein the IC comprises a System-on-Chip (SOC).
45. The at least one non-transitory computer-readable storage medium of claim 44 , wherein the hearing system further comprises a housing configured to house the SOC and a power source.
46. The at least one non-transitory computer-readable storage medium of claim 37 , wherein the method further comprises:
determining, at the controller, a signal processing path of the controller output signal as a function of specified criteria, an incoming audio signal characteristic, and/or NNE circuit feedback, wherein the signal processing path comprises the first signal processing path and/or the second signal processing path of the hearing system.
47. The at least one non-transitory computer-readable storage medium of claim 37 , wherein the method further comprises, using the NNE circuit to perform:
selectively applying a ratio mask to the controller output signal to obtain a plurality of components wherein a first component of the plurality of components corresponds to a class of sounds.
48. The at least one non-transitory computer-readable storage medium of claim 47 , wherein the method further comprises using the NNE circuit to perform:
adjusting, based on a target signal-to-noise ratio, a volume of a component of the plurality of components relative to at least one other component of the plurality of components; and
combining the plurality of components into the NNE output signal.
49. The at least one non-transitory computer-readable storage medium of claim 48 , wherein the plurality of components comprises a speech component and a noise component, and wherein the method further comprises using the NNE circuit to perform:
increasing a volume of the speech component relative to a volume of the noise component.
50. The at least one non-transitory computer-readable storage medium of claim 48 , wherein the plurality of components further comprises a user speech component and other sound source components, and wherein the method further comprises using the NNE circuit to perform:
decreasing a volume of the user speech component relative to volumes of the other sound source components.
51. The at least one non-transitory computer-readable storage medium of claim 48 , wherein the plurality of components comprises different sound source components, and wherein the method further comprises using the NNE circuit to perform:
setting respective volumes of the different sound source components.
52. The at least one non-transitory computer-readable storage medium of claim 51 , wherein the plurality of components comprises a stationary noise component and a non-stationary noise component, and wherein the method further comprises using the NNE circuit to perform:
applying a lower gain to the stationary noise component than the non-stationary noise component.
53. The at least one non-transitory computer-readable storage medium of claim 37 , wherein the second signal processing path of the hearing system excludes the NNE circuit.