IP Library › Granted Patent US 12,407,993
Granted Patent B2
US 12,407,993 · App. 18/127,518 · Granted Sep 2, 2025

Adaptive binaural filtering for listening system using remote signal sources and on-ear microphones

Inventors: Ryan M. Corey (Glen Ellyn, IL); Andrew C. Singer (Mahomet, IL)
Assignee: The Board of Trustees of the University of Illinois
H04R25/558G10K11/17827G10K11/17854H04R25/43H04R25/453H04R25/505H04R25/552H04R25/554
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,407,993
App. No.
18/127,518
Granted
Sep 2, 2025
Kind
B2
Abstract

A listening system includes one or more remote signal sources that generate one or more electronic signals, which correspond to sound in an ambient environment. A first ear microphone detects a first combination of audio signals including ambient sound and one or more propagated audio signals, corresponding to the one or more electronic signals, received at a first ear of a listener. The system includes first ear playback device and a processing device to apply a first set of audio filters including an audio filter to process a respective electronic signal of the one or more electronic signals with a first error signal, which is based on an output of the first ear microphone, to generate a first output signal to the first ear playback device, wherein acoustic cue components of the first output signal match corresponding acoustic cue components of the first combination of audio signals.

Claims (86)

1. A listening system comprising:

one or more remote signal sources that generate one or more electronic signals, which correspond to sound in an ambient environment;

a first ear microphone to detect a first combination of audio signals comprising ambient sound and one or more propagated audio signals, corresponding to the one or more electronic signals, received at a first ear of a listener;

a first ear playback device; and

a processing device coupled to the one or more remote signal sources, to the first ear microphone, and to the first ear playback device, the processing device to apply a first set of audio filters comprising an audio filter to process a respective electronic signal of the one or more electronic signals with a first error signal, which is based on an output of the first ear microphone, to generate a first output signal to the first ear playback device, wherein acoustic cue components of the first output signal match corresponding acoustic cue components of the first combination of audio signals.

2. The listening system of claim 1 , wherein the one or more remote signal sources comprises one or more microphones, an array of microphones, one or more audio signal transmitters, one or more broadcast devices, one or more sound systems, or a combination thereof.

3. The listening system of claim 1 , further comprising:

a second ear microphone to detect a second combination of audio signals comprising ambient sound and one or more propagated audio signals, corresponding to the one or more electronic signals, received at a second ear of the listener that is different than the first ear;

a second ear playback device; and

wherein the processing device is further coupled to the second ear microphone and to the second ear playback device and is further to apply a second set of audio filters comprising an audio filter to process a respective electronic signal of the one or more electronic signals with a second error signal, which is based on an output of the second ear microphone, to generate a second output signal to the second ear playback device, wherein acoustic cue components of the second output signal match corresponding acoustic cue components of the second combination of audio signals.

4. The listening system of claim 3 , further comprising:

a first hearing device containing the first ear microphone and connected to the first ear playback device; and

a second hearing device containing the second ear microphone and connected to the second ear playback device, and wherein the processing device is located within one of the first hearing device, the second hearing device, or a mobile device communicatively coupled to the first hearing device and the second hearing device.

5. The listening system of claim 3 , wherein the first set of audio filters and the second set of audio filters are defined by a parametric model that is to separately, for each of the remote signal sources, at least one of:

apply an equalization filter that is shared by both the first output signal and the second output signal; or

encode interaural time and amplitude level differences between the first output signal and the second output signal.

6. The listening system of claim 1 , wherein the first set of audio filters is defined by a parametric model, wherein the first set of audio filters is to separately, for each of the remote signal sources, at least one of:

encode a delay time for the first output signal;

perform parametric equalization for the first output signal;

encode a set of effects of ambient acoustics on a spectrum of the one or more electronic signals;

define a filter that has an impulse response of a particular length; or

define a filter described by a set of poles and zeros.

7. The listening system of claim 1 , wherein the one or more electronic signals comprise a plurality of electronic signals, each audio filter of the first set of audio filters is to generate an intermediate output signal corresponding to a respective electronic signal, and wherein the processing device is further to combine the intermediate output signals to generate the first output signal.

8. The listening system of claim 7 , wherein the processing device is further to:

process the output of the first ear microphone to generate a processed microphone signal; and

mix, into the first output signal, the processed microphone signal.

9. The listening system of claim 7 , wherein, to generate the first output signal, the processing device is further to process the combination of the intermediate output signals with post-processing that corresponds to an audio parameter.

10. The listening system of claim 9 , further comprising a user interface coupled to the processing device, wherein the processing device is further to:

receive, via the user interface, a menu selection to adjust the audio parameter; and

adjust the audio parameter of the post-processing according to the menu selection.

11. The listening system of claim 7 , wherein the first error signal comprises a difference between a combination of the intermediate output signals and the output of the first ear microphone, and wherein a single acoustic loss function of the first set of audio filters is to adaptively minimize a mean-square value or other function of the first error signal.

12. The listening system of claim 1 , wherein the audio filter comprises a first audio filter, and the first error signal comprises a difference between an output of the first audio filter and the output of the first ear microphone, and wherein the processing device is further to:

input a first electronic audio signal, from a first remote signal source, to the first audio filter;

cause the first audio filter to adaptively minimize a mean-square value or other function of the first error signal in a first intermediate output signal;

input a second electronic signal, from a second remote signal source, to a second audio filter of the first set of audio filters, wherein a second error signal comprises a difference between an output of the second audio filter and the output of the first ear microphone;

cause the second audio filter to minimize a mean-square value or other function of the second error signal in a second intermediate output signal; and

combine, to generate the first output signal, the first intermediate output signal with the second intermediate output signal.

13. The listening system of claim 12 , wherein the processing device is further to:

apply a first processing variation to the first intermediate output signal; and

apply a second processing variation to the second intermediate output signal.

14. The listening system of claim 12 , further comprising:

a first audio detector coupled to the processing device, wherein the processing device is further to disable the first audio filter in response to sound from the first remote signal source not satisfying a threshold magnitude; and

a second audio detector coupled to the processing device, wherein the processing device is further to disable the second audio filter in response to sound from the second remote signal source not satisfying the threshold magnitude.

15. A listening device comprising:

an ear microphone to detect a combination of audio signals comprising ambient sound and one or more propagated audio signals, corresponding to one or more electronic signals, received at an ear of a listener from one or more remote signal sources;

an ear playback device; and

a processing device coupled to the one or more remote signal sources, to the ear microphone, and to the ear playback device, the processing device to apply a set of audio filters comprising an audio filter to process a respective electronic signal of the one or more electronic signals with an error signal, which is based on an output of the ear microphone, to generate an output signal to the ear playback device, wherein acoustic cue components of the output signal match corresponding acoustic cue components of the combination of audio signals.

16. The listening device of claim 15 , wherein the set of audio filters is defined by a parametric model, wherein the set of audio filters is to separately, for each of the remote signal sources, at least one of:

encode a delay time for the output signal;

perform parametric equalization for the output signal;

encode a set of effects of ambient acoustics on a spectrum of the one or more electronic signals;

define a filter that has an impulse response of a particular length; or

define a filter described by a set of poles and zeros.

17. The listening device of claim 15 , wherein the one or more electronic signals comprise a plurality of electronic signals, each audio filter of the set of audio filters is to generate an intermediate output signal corresponding to a respective electronic signal, and wherein the processing device is further to combine the intermediate output signals to generate the output signal.

18. The listening device of claim 17 , wherein the processing device is further to:

process the output of the ear microphone to generate a processed microphone signal; and

mix, into the output signal, the processed microphone signal.

19. The listening device of claim 17 , wherein, to generate the output signal, the processing device is further to process the combination of the intermediate output signals with post-processing that corresponds to an audio parameter.

20. The listening device of claim 17 , wherein the error signal comprises a difference between a combination of the intermediate output signals and the output of the ear microphone, and wherein a single acoustic loss function of the set of audio filters is to adaptively minimize a mean-square value or other function of the error signal.

21. The listening device of claim 15 , wherein the audio filter comprises a first audio filter, and the error signal comprises a difference between an output of the first audio filter and the output of the ear microphone, and wherein the processing device is further to:

input a first electronic audio signal, from a first remote signal source, to the first audio filter;

cause the first audio filter to adaptively minimize a mean-square value or other function of the error signal in a first intermediate output signal;

input a second electronic signal, from a second remote signal source, to a second audio filter of the set of audio filters, wherein a second error signal comprises a difference between an output of the second audio filter and the output of the ear microphone;

cause the second audio filter to minimize a mean-square value or other function of the second error signal in a second intermediate output signal; and

combine, to generate the output signal, the first intermediate output signal with the second intermediate output signal.

22. The listening device of claim 21 , wherein the processing device is further to:

apply a first processing variation to the first intermediate output signal; and

apply a second processing variation to the second intermediate output signal.

23. The listening device of claim 21 , further comprising:

a first audio detector coupled to the processing device and to the first audio filter, wherein the processing device is further to disable the first audio filter in response to sound from the first remote signal source not satisfying a threshold magnitude; and

a second audio detector coupled to the processing device and to the second audio filter, wherein the processing device is further to disable the second audio filter in response to sound from the second remote signal source not satisfying the threshold magnitude.

24. A listening system comprising:

a first ear microphone to detect a first combination of audio signals received at a first ear of a listener, the first combination of audio signal comprising ambient sound and a plurality of propagated audio signals, corresponding to one or more electronic signals received from one or more remote microphones and one or more additional electronic signals received from one or more remote signal sources;

a first ear playback device; and

a processing device communicatively coupled to the one or more remote signal sources, to the first ear microphone, and to the first ear playback device, the processing device to apply a first set of audio filters to generate a first output signal to the first ear playback device, the first set of audio filters comprising:

at least a first audio filter to process a respective electronic signal of the one or more electronic signals with a first error signal, which is based on an output of the first ear microphone; and

at least a second audio filter to process a respective electronic signal of the one or more additional electronic signals with one of the first error signal or a second error signal based on the output of the first ear microphone; and

wherein acoustic cue components of the first output signal match corresponding acoustic cue components of the first combination of audio signals.

25. The listening system of claim 24 , further comprising the one or more remote signal sources, which comprise one or more audio signal transmitters, one or more broadcast devices, one or more sound systems, or a combination thereof.

26. The listening system of claim 24 , further comprising:

a second ear microphone to detect a second combination of audio signals comprising ambient sound and a plurality of propagated audio signals, corresponding to the one or more electronic signals and the one or more additional electronic signals, received at a second ear of the listener that is different than the first ear;

a second ear playback device; and

wherein the processing device is further coupled to the second ear microphone and to the second ear playback device and is further to apply a second set of audio filters to generate a second output signal to the second ear playback device, the second set of filters comprising:

at least a third audio filter to process a respective electronic signal of the one or more electronic signals with a third error signal, which is based on an output of the second ear microphone; and

at least a fourth audio filter to process a respective electronic signal of the one or more additional electronic signals with one of the third error signal or a fourth error signal based on the output of the second ear microphone; and

wherein acoustic cue components of the second output signal match corresponding acoustic cue components of the second combination of audio signals.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 28, 2023
From: COREY, RYAN M.; SINGER, ANDREW C.
To: THE BOARD OF TRUSTEES OF THE UNIVERSITY OF ILLINOIS
Reel/Frame 063137/0192 →
Continuity (2)
Provisional Application 63324983 · Mar 29, 2022
Related Publication 20230319492A1 · Oct 5, 2023
References Cited (53)
US 8675890B2 · Schmidt · 2014 [cited by applicant]
US 9510112B2 · Petersen · 2016 [cited by applicant]
US 10164722B1 · Napoles · 2018 [cited by examiner]
US 10182160B2 · Sorensen · 2019 [cited by applicant]
US 10438605B1 · Ganeshkumar · 2019 [cited by applicant]
US 11064302B2 · Edwards · 2021 [cited by applicant]
US 20120063610A1 · Kaulberg · 2012 [cited by applicant]
US 20190110137A1 · Udesen · 2019 [cited by applicant]
US 20200178004A1 · Courtois · 2020 [cited by applicant]
EP 3312839A1 · 2018 [cited by applicant]
EP 3333850A1 · 2018 [cited by applicant]
Agnew et al., “Just noticeable and objectionable group delays in digital hearing aids,” Journal of the American Academy of Audiology, vol. 11, No. 6, pp. 330-336, 2000. [cited by applicant]
Ali et al., “Completing the RTF vector for an MVDR beamformer as applied to a local microphone array and an external microphone,” in International Workshop on Acoustic Signal Enhancement (IWAENC), pp. 211-215, 2018. [cited by applicant]
Ali et al., “Methods of extending a generalized sidelobe canceller with external microphones,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, No. 9, pp. 1349-1364, 2019. [cited by applicant]
Araki et al., “Meeting recognition with asynchronous distributed microphone array using block-wise refinement of mask-based MVDR beamformer,” in IEEE International Conference on Acoustics, Speech and Signal Processing (… [cited by applicant]
Bertrand et al., “Robust distributed noise reduction in hearing aids with external acoustic sensor nodes,” EURASIP Journal on Advances in Signal Processing, vol. 2009, p. 530435, 2009. [cited by applicant]
Bertrand, A., “Applications and trends in wireless acoustic sensor networks: A signal processing perspective,” in IEEE Symposium on Communications and Vehicular Technology in the Benelux (SCVT), pp. 1-6, 2011. [cited by applicant]
Bogaert et al., “Speech enhancement with multichannel Wiener filter techniques in multimicrophone binaural hearing aids,” The Journal of the Acoustical Society of America, vol. 125, No. 1, pp. 360-371, 2009. [cited by applicant]
Boothroyd, A., “Hearing aid accessories for adults: The remote FM microphone,” Ear and Hearing, vol. 25, No. 1, pp. 22-33, 2004. [cited by applicant]
Corey et al., “Adaptive binaural filtering for a multiple-talker listening system using remote and on-ear microphones,” In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 1-5, 202… [cited by applicant]
Corey et al., “Binaural audio source remixing with microphone array listening devices,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 561-565, 2020. [cited by applicant]
Corey et al., “Motion-tolerant beamforming with deformable microphone arrays,” in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 115-119, 2019. [cited by applicant]
Corey, R., “Microphone array processing for augmented listening,” Ph.D. thesis, University of Illinois at Urbana-Champaign, 326 pages, 2019. [cited by applicant]
Cornelis et al., “Theoretical analysis of binaural multimicrophone noise reduction techniques,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, No. 2, pp. 342-355, 2010. [cited by applicant]
Doclo et al., “Theoretical analysis of binaural cue preservation using multi-channel Wiener filtering and interaural transfer functions,” in International Workshop on Acoustic Echo and Noise Control (IWAENC), 4 pages, 2… [cited by applicant]
Farmani et al., “Informed sound source localization using relative transfer functions for hearing aid applications,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, No. 3, pp. 611-623, 2017. [cited by applicant]
Gößling et al., “Performance analysis of the extended binaural MVDR beamformer with partial noise estimation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 462-476, 2021. [cited by applicant]
Gößling et al., “Relative transfer function estimation exploiting spatially separated microphones in a diffuse noise field,” in InternationalWorkshop on Acoustic Signal Enhancement (IWAENC), pp. 146-150, 2018. [cited by applicant]
Gößling et al., “RTF-based binaural MVDR beamformer exploiting an external microphone in a diffuse noise field,” in TG Symposium on Speech Communication, pp. 105-110, 2018. [cited by applicant]
Gößling et al., “RTF-steered binaural MVDR beamforming incorporating an external microphone for dynamic acoustic scenarios,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 416-… [cited by applicant]
Gößling et al., et al., “RTF-steered binaural MVDR beamforming incorporating multiple external microphones,” in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 373-377, 2019. [cited by applicant]
Gößling, “Binaural Beamforming Algorithms and Parameter Estimation Methods Exploiting External Microphones,” Ph. D. thesis, University of Oldenburg, 182 pages, 2020. [cited by applicant]
Hadad et al., “The binaural LCMV beamformer and its performance analysis,” IEEE Transactions on Audio, Speech and Language Processing, vol. 24, No. 3, pp. 543-558, 2016. [cited by applicant]
Hadad et al., “Theoretical analysis of binaural transfer function MVDR beamformers with interference cue preservation constraints,” IEEE Transactions on Audio, Speech and Language Processing, vol. 23, No. 12, pp. 2449-2… [cited by applicant]
Hunn, N., “Introducing Bluetooth LE Audio,” 315 pages, 2022. [cited by applicant]
Kates et al., “Integrating a remote microphone with hearing-aid processing,” The Journal of the Acoustical Society of America, vol. 145, No. 6, pp. 3551-3566, 2019. [cited by applicant]
Kates, et al., “Externalization of remote microphone signals using a structural binaural model of the head and pinna,” The Journal of the Acoustical Society of America, vol. 143, No. 5, pp. 2666-2677, 2018. [cited by applicant]
Kim et al, “A review of assistive listening device and digital wireless technology for hearing instruments,” Korean Journal of Audiology, vol. 18, No. 3, pp. 105, 2014. [cited by applicant]
Kochkin et al., “Consumer perceptions of the impact of inductively looped venues on the utility of their hearing devices,” The Hearing Review, vol. 35, No. 5, pp. 16-26, 2014. [cited by applicant]
Markovich-Golan et al., “Blind sampling rate offset estimation and compensation in wireless acoustic sensor networks with application to beamforming,” in International Workshop on Acoustic Signal Enhancement (IWAENC), 4… [cited by applicant]
Marquardt, D., “Development and evaluation of psychoacoustically motivated binaural noise reduction and cue preservation techniques,” Ph.D. dissertation, Carl von Ossietzky University of Oldenburg, 202 pages, 2016. [cited by applicant]
Marquardt, et al., “Interaural coherence preservation in multi-channel Wiener filteringbased noise reduction for binaural hearing aids,” IEEE Transactions on Audio, Speech and Language Processing, vol. 23, No. 12, pp. 2… [cited by applicant]
Stone et al., “Tolerable hearing aid delays. II. Estimation of limits imposed during speech production,” Ear and Hearing, vol. 23, No. 4, pp. 325-338, 2002. [cited by applicant]
Stone et al., “Tolerable hearing aid delays. V. Estimation of limits for open canal fittings,” Ear and hearing, vol. 29, No. 4, pp. 601-617, 2008. [cited by applicant]
Szurley et al., “Binaural noise cue preservation in a binaural noise reduction system with a remote microphone signal,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, No. 5, pp. 952-966, 2016. [cited by applicant]
Taseska et al., “Informed spatial filtering for sound extraction using distributed microphone arrays,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, No. 7, pp. 1195-1207, 2014. [cited by applicant]
Thibodeau, L., “Comparison of speech recognition with adaptive digital and FM remote microphone hearing assistance technology by listeners who use hearing aids,” American Journal of Audiology, vol. 23, No. 2, pp. 201-21… [cited by applicant]
Valimaki et al., “Assisted listening using a headset: Enhancing audio perception in real, augmented, and virtual environments,” IEEE Signal Processing Magazine, vol. 32, No. 2, pp. 92-99, 2015. [cited by applicant]
Wang et al., “Attention-based fusion for bone-conducted and air-conducted speech enhancement in the complex domain,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7757-7761, 2… [cited by applicant]
Wolfe et al., “Evaluation of performance with an adaptive digital remote microphone system and a digital remote microphone audio-streaming accessory system,” American Journal of Audiology, vol. 24, No. 3, pp. 440-450, S… [cited by applicant]
Zhu et al., “A robust speech enhancement scheme on the basis of boneconductive microphones,” in International Workshop on Signal Design and Its Applications in Communications (IWSDA), pp. 353-355, 2007. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2023/016619, mailed Jun. 27, 2023, 11 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2023/016614, mailed Jun. 28, 2023, 14 pages. [cited by applicant]