IP Library Granted Patent US 12,289,595
Granted Patent B2
US 12,289,595 · App. 17/669,560 · Granted Apr 29, 2025

Multi-channel speech compression system and method

Inventors: Dushyant Sharma (Mountain House, CA); Patrick A. Naylor (Reading, GB); Uwe Helmut Jost (Groton, MA)
Assignee: Microsoft Technology Licensing, LLC
H04S7/30G06T7/70G10L15/063G10L15/22G10L19/008G10L19/167G10L21/0208H04R1/406H04R3/005H04R5/027H04S3/008G10L2019/0001G10L2019/0002G10L2021/02166H04R2201/401H04S2400/01H04S2400/15
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,289,595
App. No.
17/669,560
Granted
Apr 29, 2025
Kind
B2
Abstract

A method, computer program product, and computing system for generating a plurality of acoustic relative transfer functions associated with a plurality of audio acquisition devices of an audio recording system deployed in an acoustic environment. Acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices may be compared. Location information associated with an acoustic source within the acoustic environment may be determined based upon, at least in part, the comparison of the acoustic relative transfer functions of the at least a pair of audio acquisition devices of the plurality of audio acquisition devices.

Claims (36)

1. A computer-implemented method, executed on a computing device, comprising:

generating a plurality of acoustic relative transfer functions associated with a microphone array having a plurality of audio acquisition devices of an audio recording system deployed in an acoustic environment, wherein each audio acquisition device of the plurality of audio acquisition devices receives a respective speech signal;

comparing acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices;

determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of the at least the pair of audio acquisition devices of the plurality of audio acquisition devices;

training a machine learning model to output the location information associated with the acoustic source by:

receiving, as input, the plurality of acoustic relative transfer functions by utilizing the machine learning model;

providing training data by correlating particular corresponding features across the plurality of acoustic relative transfer functions for a particular acoustic source to the location information associated with the acoustic source; and

generating the location information by the machine learning model based upon, at least in part, the acoustic relative transfer functions of at least the pair of audio acquisition devices of the plurality of audio acquisition devices; and

in response to determining the location information based upon, at least in part, the plurality of acoustic relative transfer functions, providing the location information associated with the acoustic source and utilizing the location information to configure or modify another device or system to enhance speaker tracking capabilities.

2. The computer-implemented method of claim 1 , wherein the plurality of audio acquisition devices of the microphone array are positioned within a fixed geometry relative to each other.

3. The computer-implemented method of claim 1 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes identifying corresponding features in the plurality of acoustic relative transfer functions of the at least a pair of audio acquisition devices of the plurality of audio acquisition devices.

4. The computer-implemented method of claim 3 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes mapping the corresponding features in the plurality of acoustic relative transfer functions of the at least a pair of audio acquisition devices of the plurality of audio acquisition devices to location information associated with the acoustic source.

5. The computer-implemented method of claim 1 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes determining the location information associated with the acoustic source within the acoustic environment using the trained machine learning model.

6. The computer-implemented method of claim 1 , wherein the location information includes one or more of azimuth information and distance information.

7. A computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising:

generating a plurality of acoustic relative transfer functions associated with a microphone array having a plurality of audio acquisition devices of an audio recording system deployed in an acoustic environment, wherein each audio acquisition device of the plurality of audio acquisition devices receives a respective speech signal;

comparing acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices;

determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of the at least the pair of audio acquisition devices of the plurality of audio acquisition devices;

training a machine learning model to output the location information associated with the acoustic source by:

receiving, as input, the plurality of acoustic relative transfer functions by utilizing the machine learning model;

providing training data by correlating particular corresponding features across the plurality of acoustic relative transfer functions for a particular acoustic source to the location information associated with the acoustic source; and

generating the location information by the machine learning model based upon, at least in part, the acoustic relative transfer functions of at least the pair of audio acquisition devices of the plurality of audio acquisition devices; and

in response to determining the location information based upon, at least in part, the plurality of acoustic relative transfer functions, providing the location information associated with the acoustic source and utilizing the location information to configure or modify another device or system to enhance speaker tracking capabilities.

8. The computer program product of claim 7 , wherein the plurality of audio acquisition devices of the audio recording system are positioned within a fixed geometry relative to each other.

9. The computer program product of claim 7 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes identifying corresponding features in the plurality of acoustic relative transfer functions of the at least a pair of audio acquisition devices of the plurality of audio acquisition devices.

10. The computer program product of claim 9 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes mapping the corresponding features in the plurality of acoustic relative transfer functions of the at least a pair of audio acquisition devices of the plurality of audio acquisition devices to location information associated with the acoustic source.

11. The computer program product of claim 7 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes determining the location information associated with the acoustic source within the acoustic environment using the trained machine learning model.

12. The computer program product of claim 7 , wherein the location information includes one or more of azimuth information and distance information.

13. A computing system comprising:

a memory; and

a processor configured to

generate a plurality of acoustic relative transfer functions associated with a microphone array having a plurality of audio acquisition devices of an audio recording system deployed in an acoustic environment, wherein each audio acquisition device of the plurality of audio acquisition devices receives a respective speech signal, wherein the processor is further configured to compare acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices, wherein the processor is further configured to determine location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of the at least the pair of audio acquisition devices of the plurality of audio acquisition devices, wherein the processor is further configured to train a machine learning model to output the location information associated with the acoustic source by: receiving, as input, the plurality of acoustic relative transfer functions by utilizing the machine learning model, providing training data by correlating particular corresponding features across the plurality of acoustic relative transfer functions for a particular acoustic source to the location information associated with the acoustic source, and generating the location information by the machine learning model based upon, at least in part, the acoustic relative transfer functions of at least the pair of audio acquisition devices of the plurality of audio acquisition devices, and wherein in response to determining the location information based upon, at least in part, the plurality of acoustic relative transfer functions, the processor is further configured to provide the location information associated with the acoustic source and utilizing the location information to configure or modify another device or system to enhance speaker tracking capabilities.

14. The computing system of claim 13 , wherein the plurality of audio acquisition devices of the audio recording system are positioned within a fixed geometry relative to each other.

15. The computing system of claim 13 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes identifying corresponding features in the plurality of acoustic relative transfer functions of the at least a pair of audio acquisition devices of the plurality of audio acquisition devices.

16. The computing system of claim 15 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes mapping the corresponding features in the plurality of acoustic relative transfer functions of the at least a pair of audio acquisition devices of the plurality of audio acquisition devices to location information associated with the acoustic source.

17. The computing system of claim 13 , wherein determining location information associated with an acoustic source within the acoustic environment based upon, at least in part, the comparison of the acoustic relative transfer functions of at least a pair of audio acquisition devices of the plurality of audio acquisition devices includes determining the location information associated with the acoustic source within the acoustic environment using the trained machine learning model.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 14, 2023
From: NUANCE COMMUNICATIONS, INC.
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 065578/0676 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 10, 2022
From: SHARMA, DUSHYANT; NAYLOR, PATRICK A.; JOST, UWE HELMUT
To: NUANCE COMMUNICATIONS, INC.
Reel/Frame 059224/0746 →
Continuity (3)
Provisional Application 63148427 · Feb 11, 2021
Provisional Application 63183848 · May 4, 2021
Related Publication 20220254357A1 · Aug 11, 2022
References Cited (112)
US 5444816A · Adoul et al. · 1995 [cited by applicant]
US 6707918B1 · Mcgrath et al. · 2004 [cited by applicant]
US RE40054E · Girod · 2008 [cited by applicant]
US 9992587B2 · Farmani et al. · 2018 [cited by applicant]
US 10219083B2 · Farmani et al. · 2019 [cited by applicant]
US 10373612B2 · Parthasarathi et al. · 2019 [cited by applicant]
US 10390171B2 · Lee et al. · 2019 [cited by applicant]
US 10490205B1 · Iyengar · 2019 [cited by applicant]
US 10598543B1 · Mansour et al. · 2020 [cited by applicant]
US 10638252B1 · Donley · 2020 [cited by examiner]
US 11234073B1 · Xu · 2022 [cited by applicant]
US 20060217977A1 · Gaeta · 2006 [cited by applicant]
US 20100324890A1 · Adeney · 2010 [cited by examiner]
US 20110087491A1 · Wittenstein et al. · 2011 [cited by applicant]
US 20120163606A1 · Eronen et al. · 2012 [cited by applicant]
US 20130022189A1 · Ganong, III · 2013 [cited by examiner]
US 20130151249A1 · Nakadai et al. · 2013 [cited by applicant]
US 20130162821A1 · Park et al. · 2013 [cited by applicant]
US 20130253923A1 · Mustiere · 2013 [cited by applicant]
US 20130287225A1 · Niwa et al. · 2013 [cited by applicant]
US 20140078867A1 · Nakamura · 2014 [cited by applicant]
US 20150189455A1 · Donaldson · 2015 [cited by applicant]
US 20170353789A1 · Kim et al. · 2017 [cited by applicant]
US 20180007488A1 · Horowitz et al. · 2018 [cited by applicant]
US 20180249275A1 · Ghorbal · 2018 [cited by applicant]
US 20180255418A1 · Tzirkel-Hancock · 2018 [cited by examiner]
US 20180262849A1 · Farmani et al. · 2018 [cited by applicant]
US 20180359596A1 · Breebaart et al. · 2018 [cited by applicant]
US 20190122686A1 · Slapak · 2019 [cited by examiner]
US 20190172450A1 · Mustiere et al. · 2019 [cited by applicant]
US 20190208351A1 · Lyren et al. · 2019 [cited by applicant]
US 20190212441A1 · Casner et al. · 2019 [cited by applicant]
US 20190219660A1 · Cordourier Maruri et al. · 2019 [cited by applicant]
US 20190311730A1 · Looney et al. · 2019 [cited by applicant]
US 20190355380A1 · Hatfield · 2019 [cited by applicant]
US 20190394564A1 · Mehra et al. · 2019 [cited by applicant]
US 20190394565A1 · Mehra · 2019 [cited by applicant]
US 20200035231A1 · Parthasarathi et al. · 2020 [cited by applicant]
US 20200051565A1 · Singh · 2020 [cited by applicant]
US 20200058310A1 · Goodwin · 2020 [cited by applicant]
US 20200077185A1 · Nakadai et al. · 2020 [cited by applicant]
US 20200107149A1 · Sunder · 2020 [cited by applicant]
US 20200118547A1 · Weng et al. · 2020 [cited by applicant]
US 20200304933A1 · Lien et al. · 2020 [cited by applicant]
US 20200342857A1 · Moreno et al. · 2020 [cited by applicant]
US 20200372897A1 · Battenberg et al. · 2020 [cited by applicant]
US 20200374646A1 · Lee et al. · 2020 [cited by applicant]
US 20200395003A1 · Sharma et al. · 2020 [cited by applicant]
US 20200410045A1 · Vozila et al. · 2020 [cited by applicant]
US 20210076150A1 · Ithapu et al. · 2021 [cited by applicant]
US 20210158086A1 · Salamon · 2021 [cited by examiner]
US 20210160605A1 · Igarashi · 2021 [cited by applicant]
US 20210211829A1 · Riggs et al. · 2021 [cited by applicant]
US 20210235189A1 · Secall · 2021 [cited by applicant]
US 20210235213A1 · Lee · 2021 [cited by applicant]
US 20220254356A1 · Sharma et al. · 2022 [cited by applicant]
US 20220254358A1 · Sharma et al. · 2022 [cited by applicant]
US 20220254359A1 · Sharma et al. · 2022 [cited by applicant]
US 20220254360A1 · Sharma et al. · 2022 [cited by applicant]
US 20220254361A1 · Sharma et al. · 2022 [cited by applicant]
US 20220256303A1 · Sharma et al. · 2022 [cited by applicant]
US 20220345813A1 · Stein · 2022 [cited by applicant]
US 20230054213A1 · De Haan · 2023 [cited by applicant]
US 20240323630A1 · Sharma · 2024 [cited by applicant]
CN 112346012A · 2021 [cited by applicant]
EP 3413589A1 · 2018 [cited by applicant]
WO 2007011157A1 · 2007 [cited by applicant]
WO 2020089523A1 · 2020 [cited by applicant]
WO 2020231883A1 · 2020 [cited by applicant]
WO 2021071498A1 · 2021 [cited by applicant]
Hu et al, Semi-Supervised Multiple Source Localization Using Relative Harmonic Coefficients Under Noisy and Reverberant Environments, 2020, IEEE, p. 3108 and 3109 (Year: 2020). [cited by examiner]
Opochinksy et al., Deep-Ranking Based Sound Source Localization, 2019, IEEE, whole document (Year: 2019). [cited by examiner]
Non-Final office Action mailed on Mar. 18, 2024, in U.S. Appl. No. 17/669,599, 9 pages. [cited by applicant]
Notice of Allowance mailed on Feb. 5, 2024, in U.S. Appl. No. 17/669,574, 2 pages. [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 17/669,592”, Mailed Date: Apr. 12, 2023, 18 Pages. [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 17/669,592”, Mailed Date: May 4, 2023, 20 Pages. [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 17/669,534”, Mailed Date: Nov. 2, 2023, 10 Pages. [cited by applicant]
“Notice of Allowance Issued in U.S. Appl. No. 17/669,574”, Mailed Date: Nov. 27, 2023, 8 Pages. [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 17/669,574”, Mailed Date: Aug. 11, 2023, 10 Pages. [cited by applicant]
“Notice of Allowance Issued in U.S. Appl. No. 17/669,549”, Mailed Date: Aug. 29, 2023, 7 Pages. [cited by applicant]
“Final Office Action Issued in U.S. Appl. No. 17/669,592”, Mailed Date: Sep. 26, 2023, 22 Pages. [cited by applicant]
Non-Final Office Action mailed on Jan. 4, 2024, in U.S. Appl. No. 17/669,567, 11 pages. [cited by applicant]
Notice of Allowance mailed on Jan. 3, 2024, in U.S. Appl. No. 17/669,574, 2 pages. [cited by applicant]
Notice of Allowance mailed on Jan. 17, 2024, in U.S. Appl. No. 17/669,592, 12 pages. [cited by applicant]
Laufer, et al., “Relative Transfer Function Modeling for Supervised Source Localization”, In Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Oct. 20, 2013, 4 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/016021”, Mailed Date: Jun. 22, 2022, 8 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/016024”, Mailed Date: May 12, 2022, 11 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/016027”, Mailed Date: May 26, 2022, 9 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/016030”, Mailed Date: May 12, 2022, 9 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/016032”, Mailed Date: Jul. 5, 2022, 11 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/016033”, Mailed Date: Jun. 8, 2022, 11 Pages. [cited by applicant]
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US22/016034”, Mailed Date: Jun. 3, 2022, 10 Pages. [cited by applicant]
Notice of Allowance mailed on May 22, 2024, in U.S. Appl. No. 17/669,567 8 pages. [cited by applicant]
Notice of Allowance mailed on Jul. 11, 2024, in U.S. Appl. No. 17/669,599, 14 pages. [cited by applicant]
Notice of Allowance mailed on Apr. 17, 2024, in U.S. Appl. No. 17/669,534, 08 pages. [cited by applicant]
Extended European Search Report received for EP Application No. 22753366.8, mailed on Dec. 4, 2024, 6 pages. [cited by applicant]
Extended European Search Report Received for European Application No. 22753372.6, mailed on Dec. 3, 2024, 07 pages. [cited by applicant]
Extended Search Report Received for European Application No. 22753368.4, mailed on Nov. 26, 2024, 08 pages. [cited by applicant]
Jones, et al., “Spatial Coding for Microphone Arrays using IPNLMS-Based RTF Estimation”, IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Oct. 2021, pp. 76-80. [cited by applicant]
Qiang et al., “Multi-Channel Speech Coding Combining Spatial Information and Speex Codec”, 2014 12th International Conference on Signal Processing (ICSP), IEEE, vol. 1, Dec. 6, 2020, pp. 136-140. [cited by applicant]
Yutong, et al., “Accounting for Room Acoustics in Audio-Visual Multi-Speaker Tracking”, 2018 IEEE International Conference On Acoustics, Speech and Signal Processing (ICASSP), Apr. 15, 2018, pp. 6553-6557. [cited by applicant]
Extended European search report received for European Patent Application No. 22753370.0, mailed on Jan. 23, 2025, 10 pages. [cited by applicant]
Farmani, et al., “Informed Sound Source Localization Using Relative Transfer Functions for Hearing Aid Applications”, IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, Issue 3, Jan. 10, 2017, pp.… [cited by applicant]
U.S. Appl. No. 17/669,534, filed Feb. 11, 2022. [cited by applicant]
U.S. Appl. No. 17/669,549, filed Feb. 11, 2022. [cited by applicant]
U.S. Appl. No. 17/669,567, filed Feb. 11, 2022. [cited by applicant]
U.S. Appl. No. 17/669,574, filed Feb. 11, 2022. [cited by applicant]
U.S. Appl. No. 17/669,592, filed Feb. 11, 2022. [cited by applicant]
U.S. Appl. No. 17/669,599, filed Feb. 11, 2022. [cited by applicant]
Communication pursuant to Rules 70(2) and 70a(2) Received for European Application No. 22753370.0, mailed on Feb. 11, 2025, 1 page. [cited by applicant]
Extended European Search Report for European Application No. 22753376.7, mailed on Dec. 4, 2024, 13 pages. [cited by applicant]
Non-Final Office Action mailed on Feb. 28, 2025, in U.S. Appl. No. 18/676,347, 21 Pages. [cited by applicant]