Method, apparatus and system for neural network enabled hearing aid
The disclosure generally relates to a method, system and apparatus to improve a user's understanding of speech in real-time conversations by processing the audio through a neural network contained in a hearing device. The hearing device may be a headphone or hearing aid. In one embodiment, the disclosure relates to an apparatus to enhance incoming audio signal. The apparatus includes a controller to receive an incoming signal and provide a controller output signal; a neural network engine (NNE) circuitry in communication with the controller, the NNE circuitry activatable by the controller, the NNE circuitry configured to generate an NNE output signal from the controller output signal; and a digital signal processing (DSP) circuitry to receive one or more of controller output signal or the NNE circuitry output signal to thereby generate a processed signal; wherein the controller determines a processing path of the controller output signal through one of the DSP or the NNE circuitries as a function of one or more of predefined parameters, incoming signal characteristics and NNE circuitry feedback.
1. An ear-worn device configured to enhance incoming audio signals, the ear-worn device comprising:
neural network circuitry configured to denoise an incoming audio signal by:
using a neural network to separate a speech component of the incoming audio signal from a noise component of the incoming audio signal; and
mixing the speech component of the incoming audio signal with an amount of the noise component of the incoming audio signal;
digital signal processing circuitry coupled to the neural network circuitry and configured to perform one or more of dynamic range compression, amplification, and frequency tuning; and
a controller configured to selectively transmit the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry.
2. The ear-worn device of claim 1 , wherein the neural network circuitry is configured, when denoising the incoming audio signal, to apply a level of denoising that is less than a maximum level of denoising achievable by the neural network circuitry.
3. The ear-worn device of claim 2 , wherein:
the level of denoising that is less than the maximum level of denoising achievable by the neural network circuitry is a first level of denoising;
the controller is configured to determine whether a metric characterizing an aspect of an acoustic environment of the ear-worn device satisfies at least one criterion; and
based on the controller determining that the metric characterizing the aspect of the acoustic environment of the ear-worn device satisfies the at least one criterion, the neural network circuitry is configured to denoise the incoming audio signal by applying a second level of denoising that is greater than the first level of denoising.
4. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to determine whether a user selection of an operating mode through an application on a smartphone has been received.
5. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to determine whether a user selection of an input on the ear-worn device has been received.
6. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to:
detect a signal-to-noise ratio (SNR) for the incoming audio signal; and
compare the detected SNR with a threshold SNR.
7. The ear-worn device of claim 6 , wherein the controller is further configured to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry if the detected SNR is above the threshold SNR.
8. The ear-worn device of claim 6 , wherein the controller is further configured to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry if the detected SNR is below the threshold SNR.
9. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to:
detect a signal-to-noise ratio (SNR) for the incoming audio signal;
compare the detected SNR with a first threshold SNR and a second threshold SNR; and
determine to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry if the detected SNR is above the first threshold SNR; or below the second threshold SNR.
10. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to determine a performance metric indicative of model confidence.
11. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to detect a period of silence.
12. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to determine a battery level of the ear-worn device.
13. The ear-worn device of claim 1 , wherein the controller is configured, when selectively transmitting the incoming audio signal to the neural network circuitry for denoising or to transmit the incoming audio signal to the digital signal processing circuitry without denoising by the neural network circuitry, to determine voice activity using a voice activity detector.
14. The ear-worn device of claim 1 , wherein the ear-worn device is further configured to perform a short-time Fourier transform on the incoming audio signal prior to denoising by the neural network circuitry.
15. The ear-worn device of claim 14 , wherein computation by the neural network circuitry and the digital signal processing circuitry completes in less time than a time window of the short-time Fourier transform.
16. The ear-worn device of claim 1 , wherein the neural network circuitry is integrated on an integrated circuit in the ear-worn device.
17. The ear-worn device of claim 16 , wherein the digital signal processing circuitry is integrated on a different core than the neural network circuitry.
18. The ear-worn device of claim 1 , further comprising an accelerometer, and wherein the neural network circuitry is configured to use acceleration data from the accelerometer for inference.
19. The ear-worn device of claim 1 , wherein:
the neural network circuitry is configured to denoise the incoming audio signal by:
generating a mask based on the incoming audio signal; and
applying the mask to the incoming audio signal such that the speech component of the incoming audio signal is obtained.
20. The ear-worn device of claim 19 , wherein the neural network circuitry is configured to determine the noise component of the incoming audio signal by:
generating a second mask based on the incoming audio signal and applying the second mask to the incoming audio signal such that the noise component of the incoming audio signal is obtained; or
subtracting the speech component of the incoming audio signal from the incoming audio signal.