IP Library Granted Patent US 12,452,620
Granted Patent B2
US 12,452,620 · App. 18/676,347 · Granted Oct 21, 2025

Multi-channel speech compression system and method

Inventors: Dushyant Sharma (Mountain House, CA); Patrick A. Naylor (Reading, GB); Uwe Helmut Jost (Groton, MA)
Assignee: Microsoft Technology Licensing, LLC
H04S7/30G06T7/70G10L15/063G10L15/22G10L19/008G10L19/167G10L21/0208H04R1/406H04R3/005H04R5/027H04S3/008G10L2019/0001G10L2019/0002G10L2021/02166H04R2201/401H04S2400/01H04S2400/15
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,452,620
App. No.
18/676,347
Granted
Oct 21, 2025
Kind
B2
Abstract

A method, computer program product, and computing system for encoding audio encounter information of a reference audio acquisition device of a plurality of audio acquisition devices of an audio recording system, thus defining encoded reference audio encounter information. Location information may be estimated, via a machine vision system, for an acoustic source within an acoustic environment. One or more acoustic relative transfer functions may be selected from a plurality of acoustic relative transfer functions for the plurality of audio acquisition devices of the audio recording system based upon, at least in part, the location information. The encoded reference audio encounter information and a representation of the selected one or more acoustic relative transfer function may be transmitted.

Claims (40)

1. A computer-implemented method, executed on a computing device, comprising:

encoding audio encounter information of a reference audio acquisition device of a plurality of audio acquisition devices of an audio recording system, thus defining encoded reference audio encounter information;

estimating, via a machine vision system, location information for an acoustic source within an acoustic environment;

accessing an acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system, wherein the acoustic relative transfer function codebook includes a plurality of acoustic relative transfer functions between the reference audio acquisition device and the plurality of audio acquisition devices of the audio recording system, wherein each of the plurality of acoustic relative transfer functions is associated with a unique identifier for being uniquely identifiable from the plurality of acoustic relative transfer functions of the acoustic relative transfer function codebook based upon, at least in part, the location information and speakers within the acoustic environment, and wherein the acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system includes a data structure configured to store mapping characteristics for mapping speech signals obtained by the reference audio acquisition device to speech signals of another audio acquisition device of the plurality of audio acquisition devices;

selecting one or more acoustic relative transfer functions from the acoustic relative transfer function codebook based upon, at least in part, the location information; and

transmitting the encoded reference audio encounter information and a representation of the selected one or more acoustic relative transfer functions.

2. The computer-implemented method of claim 1 , wherein the plurality of audio acquisition devices of the audio recording system are positioned within a fixed geometry relative to each other.

3. The computer-implemented method of claim 1 , further comprising:

mapping the location information for the acoustic source within the acoustic environment to the plurality of acoustic relative transfer functions, wherein the one or more acoustic relative transfer functions are selected from the acoustic relative transfer function codebook based on the mapping of the location information to the plurality of acoustic relative transfer functions.

4. The computer-implemented method of claim 1 , wherein accessing the acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system includes generating a plurality of residual signals associated with the selected one or more acoustic relative transfer functions.

5. The computer-implemented method of claim 1 , wherein the representation of the selected one or more acoustic relative transfer functions includes a vector generated by encoding the selected one or more acoustic relative transfer functions.

6. The computer-implemented method of claim 1 , wherein the representation of the selected one or more acoustic relative transfer functions includes a vector generated by encoding one or more codebook entries within the acoustic relative transfer function codebook, the one or more codebook entries corresponding to the selected one or more acoustic relative transfer functions.

7. The computer-implemented method of claim 1 , wherein the machine vision system includes red, green, blue (RGB) sensors.

8. The computer-implemented method of claim 1 , wherein the machine vision system includes infrared sensors.

9. The computer-implemented method of claim 1 , wherein the machine vision system includes ultraviolet sensors.

10. The computer-implemented method of claim 1 , wherein the machine vision system includes SONAR sensors.

11. The computer-implemented method of claim 1 , wherein the machine vision system includes RADAR sensors.

12. The computer-implemented method of claim 1 , wherein the machine vision system includes thermal imaging sensors.

13. A non-transitory computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising:

encoding audio encounter information of a reference audio acquisition device of a plurality of audio acquisition devices of an audio recording system, thus defining encoded reference audio encounter information;

estimating, via a machine vision system, location information for an acoustic source within an acoustic environment;

accessing an acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system, wherein the acoustic relative transfer function codebook includes a plurality of acoustic relative transfer functions between the reference audio acquisition device and the plurality of audio acquisition devices of the audio recording system, wherein each of the plurality of acoustic relative transfer functions is associated with a unique identifier for being uniquely identifiable from the plurality of acoustic relative transfer functions of the acoustic relative transfer function codebook based upon, at least in part, the location information and speakers within the acoustic environment, wherein the acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system includes a data structure configured to store mapping characteristics for mapping speech signals obtained by the reference audio acquisition device to speech signals of another audio acquisition device of the plurality of audio acquisition devices;

selecting one or more acoustic relative transfer functions from the acoustic relative transfer function codebook based upon, at least in part, the location information; and

transmitting the encoded reference audio encounter information and a representation of the selected one or more acoustic relative transfer functions.

14. The non-transitory computer program product of claim 13 , wherein the plurality of audio acquisition devices of the audio recording system are positioned within a fixed geometry relative to each other.

15. The non-transitory computer program product of claim 13 , wherein the operations further comprise:

mapping the location information for the acoustic source within the acoustic environment to the plurality of acoustic relative transfer functions, wherein the one or more acoustic relative transfer functions are selected from the acoustic relative transfer function codebook based on the mapping of the location information to the plurality of acoustic relative transfer functions.

16. The non-transitory computer program product of claim 13 , wherein accessing the acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system includes generating a plurality of residual signals associated with the selected one or more acoustic relative transfer functions.

17. A computing system comprising:

a memory storing programming instructions; and

a processor configured to execute the programming instructions stored in the memory, wherein the programming instructions, upon execution by the processor, cause the system to:

encode audio encounter information of a reference audio acquisition device of a plurality of audio acquisition devices of an audio recording system, thus defining encoded reference audio encounter information;

estimate, via a machine vision system, location information for an acoustic source within an acoustic environment;

access an acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system, wherein the acoustic relative transfer function codebook includes a plurality of acoustic relative transfer functions between the reference audio acquisition device and the plurality of audio acquisition devices of the audio recording system, wherein each of the plurality of acoustic relative transfer functions is associated with a unique identifier for being uniquely identifiable from the plurality of acoustic relative transfer functions of the acoustic relative transfer function codebook based upon, at least in part, the location information and speakers within the acoustic environment, and wherein the acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system includes a data structure configured to store mapping characteristics for mapping speech signals obtained by the reference audio acquisition device to speech signals of another audio acquisition device of the plurality of audio acquisition devices;

select one or more acoustic relative transfer functions from the acoustic relative transfer function codebook based upon, at least in part, the location information; and

transmit the encoded reference audio encounter information and the selected one or more acoustic relative transfer functions corresponding to the encoded reference audio information.

18. The computing system of claim 17 , wherein the plurality of audio acquisition devices of the audio recording system are positioned within a fixed geometry relative to each other.

19. The computing system of claim 17 , wherein the processor is further configured to:

map the location information for the acoustic source within the acoustic environment to the plurality of acoustic relative transfer functions, wherein the one or more acoustic relative transfer functions are selected from the acoustic relative transfer function codebook based on the mapping of the location information to the plurality of acoustic relative transfer functions.

20. The computing system of claim 17 , wherein accessing the acoustic relative transfer function codebook for the plurality of audio acquisition devices of the audio recording system includes generating a plurality of residual signals associated with the selected one or more acoustic relative transfer functions.

Continuity (4)
Continuation 17669592 · Feb 11, 2022
Provisional Application 63183848 · May 4, 2021
Provisional Application 63148427 · Feb 11, 2021
Related Publication 20240323630A1 · Sep 26, 2024
References Cited (39)
US 10490205B1 · Iyengar · 2019 [cited by examiner]
US 10638252B1 · Donley · 2020 [cited by applicant]
US 11234073B1 · Xu · 2022 [cited by examiner]
US 20100027805A1 · Itou · 2010 [cited by examiner]
US 20110087491A1 · Wittenstein · 2011 [cited by examiner]
US 20130151249A1 · Nakadai et al. · 2013 [cited by applicant]
US 20130162821A1 · Park · 2013 [cited by applicant]
US 20140078867A1 · Nakamura · 2014 [cited by applicant]
US 20180249275A1 · Ghorbal · 2018 [cited by examiner]
US 20190311730A1 · Looney et al. · 2019 [cited by applicant]
US 20190394565A1 · Mehra · 2019 [cited by applicant]
US 20200077185A1 · Nakadai · 2020 [cited by examiner]
US 20200107149A1 · Sunder · 2020 [cited by examiner]
US 20210235189A1 · Secall · 2021 [cited by examiner]
US 20220345813A1 · Stein · 2022 [cited by applicant]
US 20230054213A1 · De Haan · 2023 [cited by examiner]
CN 112346012A · 2021 [cited by applicant]
EP 3413589A1 · 2018 [cited by applicant]
WO 2007011157A1 · 2007 [cited by applicant]
WO 2020089523A1 · 2020 [cited by applicant]
WO 2020231883A1 · 2020 [cited by applicant]
Notice of Allowance mailed on Oct. 29, 2024, in U.S. Appl. No. 17/669,560, 10 pages. [cited by applicant]
Notice of Allowance mailed on Apr. 17, 2024, in U.S. Appl. No. 17/669,534, 08 pages. [cited by applicant]
Notice of Allowance mailed on Jul. 11, 2024, in U.S. Appl. No. 17/669,599, 14 pages. [cited by applicant]
Communication pursuant to Rules 70(2) and 70a (2) received in European Application No. 22753366.8, mailed on Jan. 2, 2025, 1 page. [cited by applicant]
Communication pursuant to Rules 70(2) and 70a (2) received in European Application No. 22753376.7, mailed on Jan. 2, 2025, 1 page. [cited by applicant]
Communication pursuant to Rules 70(2) and 70a(2) Received for European Application No. 22753368.4, mailed on Dec. 13, 2024, 1 Page. [cited by applicant]
Communication pursuant to Rules 70(2) and 70a(2) Received for European Application No. 22753370.0, mailed on Feb. 11, 2025, 1 page. [cited by applicant]
Communication pursuant to Rules 70(2) and 70a(2) received in European Application No. 22753372.6, mailed on Dec. 20, 2024, 1 Page. [cited by applicant]
Extended European Search Report for European Application No. 22753376.7, mailed on Dec. 4, 2024, 13 pages. [cited by applicant]
Extended European Search Report received for EP Application No. 22753366.8, mailed on Dec. 4, 2024, 6 pages. [cited by applicant]
Extended European Search Report Received for European Application No. 22753372.6, mailed on Dec. 3, 2024, 07 pages. [cited by applicant]
Extended European search report received for European Patent Application No. 22753370.0, mailed on Jan. 23, 2025, 10 pages. [cited by applicant]
Extended Search Report Received for European Application No. 22753368.4, mailed on Nov. 26, 2024, 08 pages. [cited by applicant]
Farmani, et al., “Informed Sound Source Localization Using Relative Transfer Functions for Hearing Aid Applications”, IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, Issue 3, Jan. 10, 2017, pp.… [cited by applicant]
Jones, et al., “Spatial Coding for Microphone Arrays using IPNLMS-Based RTF Estimation”, IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Oct. 2021, pp. 76-80. [cited by applicant]
Qiang et al., “Multi-Channel Speech Coding Combining Spatial Information and Speex Codec”, 2014 12th International Conference on Signal Processing (ICSP), IEEE, vol. 1, Dec. 6, 2020, pp. 136-140. [cited by applicant]
Supplemental Notice of Allowance mailed on Feb. 5, 2025, in U.S. Appl. No. 17/669,560, 04 pages. [cited by applicant]
Yutong, et al., “Accounting for Room Acoustics in Audio-Visual Multi-Speaker Tracking”, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 15, 2018, pp. 6553-6557. [cited by applicant]