Method, apparatus and system for low latency audio enhancement
According to some embodiments, an ear-worn device, e.g., a hearing aid, is provided that operates to enhance audio signals detected by the ear-worn device. In some embodiments, the ear-worn device includes a microphone, a processing circuit coupled to the microphone, and an output signal generator coupled to the processing circuit. In some embodiments, a method for enhancing audio signals includes: detecting an audio signal with the microphone; as the audio signal is being detected, dividing the audio signal into a plurality of segments; enhancing the detected audio signal with the processing circuit of the hearing aid; and outputting the enhanced audio signal with the output signal generator. In some embodiments, enhancing the detected audio signal includes processing one or more of the segments of the detected audio signal with a neural network engine (NNE) of the processing circuit to obtain an output for enhancing the detected audio signal.
1. A method for enhancing audio signals with a hearing aid, the hearing aid comprising a microphone, a processing circuit coupled to the microphone, and an output signal generator coupled to the processing circuit, the method comprising:
detecting an audio signal with the microphone of the hearing aid;
as the audio signal is being detected, dividing, with the processing circuit of the hearing aid, the audio signal into a plurality of overlapping segments, the plurality of overlapping segments comprising a first segment and a second segment, the first segment and the second segment sharing an overlapping portion;
after detecting the first segment, enhancing the overlapping portion by processing the first segment with a neural network engine (NNE) of the processing circuit to obtain a first output for enhancing the first segment including the overlapping portion;
transmitting the enhanced overlapping portion to the output signal generator for playback;
beginning playback of the enhanced overlapping portion with the output signal generator;
detecting the second segment during the playback of the enhanced overlapping portion;
after detecting the second segment including the overlapping portion and a non-overlapping portion, enhancing the non-overlapping portion of the second segment by processing the second segment with the NNE to obtain a second output for enhancing the second segment including the overlapping portion and the non-overlapping portion;
transmitting the enhanced non-overlapping portion to the output signal generator for playback; and
beginning playback of the enhanced non-overlapping portion with the output signal generator.
2. The method of claim 1 , wherein enhancing the overlapping portion comprises enhancing the overlapping portion with a digital signal processing (DSP) circuit of the processing circuit of the hearing aid.
3. The method of claim 1 ,
wherein processing the first segment with the NNE of the processing circuit to obtain the first output comprises isolating a first speech component of the overlapping portion from a first noise component of the overlapping portion, and
wherein processing the second segment with the NNE comprises isolating a second speech component of the overlapping portion from a second noise component of the overlapping portion.
4. The method of claim 1 , wherein enhancing the overlapping portion by processing the first segment with the NNE to obtain the first output for enhancing the first segment comprises:
generating a mask for the first segment including the overlapping portion;
applying the mask to the first segment to generate an enhanced first segment including the enhanced overlapping portion and an enhanced non-overlapping portion; and
excluding the enhanced non-overlapping portion from further processing.
5. The method of claim 1 , wherein a first length of the first segment is longer than the overlapping portion, and wherein a second length of the second segment is longer than the overlapping portion.
6. The method of claim 1 ,
wherein the plurality of overlapping segments further comprises a third segment, the third segment and the second segment sharing a second overlapping portion, the second overlapping portion comprising the non-overlapping portion of the second segment, and wherein the method further comprises:
detecting the third segment during the playback of the enhanced non-overlapping portion of the second segment;
after detecting the third segment including the second overlapping portion and a non-overlapping portion, enhancing the non-overlapping portion of the third segment by processing the third segment with the NNE to obtain a third output for enhancing the third segment including the non-overlapping portion;
transmitting the enhanced non-overlapping portion of the third segment to the output signal generator for playback; and
beginning playback of the enhanced non-overlapping portion of the third segment with the output signal generator.
7. The method of claim 1 , wherein the plurality of overlapping segments further comprises a third segment, the method further comprising:
selectively transmitting, with a controller of the processing circuit of the hearing aid, the third segment to the NNE or a digital signal processing (DSP) circuit based on a characteristic of the third segment.
8. The method of claim 7 , wherein a character of the third segment comprises a noise level of the third segment compared to a threshold noise level, and wherein selectively transmitting the third segment to the NNE or the DSP circuit based on the characteristic of the third segment comprises transmitting the third segment to the DSP circuit when the noise level is below the threshold noise level.
9. The method of claim 1 , wherein the NNE comprises a recurrent NNE.
10. A method for enhancing audio signals with a hearing aid, the hearing aid comprising a microphone, a processing circuit coupled to the microphone, and an output signal generator coupled to the processing circuit, the method comprising:
detecting an audio signal with the microphone of the hearing aid;
as the audio signal is being detected, dividing, with the processing circuit of the hearing aid, the audio signal into a plurality of segments, the plurality of segments comprising a first segment and a second segment, the first segment preceding the second segment in time;
after detecting the first segment, processing the first segment with a neural network engine (NNE) to obtain a first output for enhancing the second segment;
enhancing the second segment based on the first output of the NNE;
transmitting the enhanced second segment to an output signal generator for playback; and
beginning playback of the enhanced second segment with the output signal generator.
11. The method of claim 10 , wherein dividing the detected audio signal into the plurality of segments comprises dividing the detected audio signal using a controller of the processing circuit, and wherein enhancing the second segment with the processing circuit of the hearing aid further comprises:
receiving, at a digital signal processing (DSP) circuit, the second segment from the controller, the receiving beginning at an initial time,
wherein enhancing the second segment based on the first output of the NNE comprises enhancing the second segment with the DSP circuit of the processing circuit.
12. The method of claim 11 , wherein enhancing the second segment with the DSP circuit comprises enhancing the second segment with the DSP circuit after an elapsed time since the initial time, the elapsed being equivalent to a length of the second segment.
13. The method of claim 11 , wherein enhancing the second segment with the DSP circuit comprises enhancing the second segment with the DSP circuit after an elapsed time since the initial time, the elapsed time being equal to or shorter than a duration of processing the first segment with the NNE to obtain the first output for enhancing the detected audio signal.
14. The method of claim 10 , wherein the plurality of segments further comprises a third segment, the first segment and the second segment preceding the third segment in time, the method further comprising:
after detecting the second segment, processing the second segment with the NNE to obtain a second output for enhancing the third segment; and
enhancing the third segment based on the second output of the NNE.
15. The method of claim 10 , wherein the plurality of segments further comprises a third segment, the first segment and the second segment preceding the third segment in time, the method further comprising:
prior to completing the processing of the second segment with the NNE, detecting the third segment;
enhancing the third segment based on the first output of the NNE; and
transmitting the enhanced third segment to the output signal generator for playback.
16. The method of claim 10 ,
wherein processing the first segment with the NNE to obtain the first output for enhancing the second segment comprises generating, based on the first segment, a mask for the second segment, and
wherein enhancing the second segment based on the first output of the NNE comprise applying the generated mask to the second segment.
17. The method of claim 10 , wherein the NNE comprises a recurrent NNE.
18. The method of claim 10 , wherein detecting the first segment comprises detecting the first segment during a first time period, and wherein detecting the second segment comprises detecting the second segment beginning at a time after the first time period.