Granted Patent B1

US 9,865,274 · App. 15/388,552 · Granted Jan 9, 2018

Abstract

Claims (120)

1. A method of processing ambisonic audio to provide three dimensional audio during a bidirectional real-time communication session between local meeting equipment and remote meeting equipment, comprising:

receiving, during the bidirectional real-time communication session, an input ambisonic audio signal, the input ambisonic audio signal including a plurality of channels, wherein each channel in the input ambisonic audio signal is made up of audio data representing sound captured by an ambisonic microphone located in the local meeting equipment;

receiving a remote audio signal, wherein the remote audio signal is made up of audio data representing sound captured by a microphone contained in remote meeting equipment, and wherein the remote audio signal is passed to at least one local loudspeaker located in the local meeting equipment for output by the local loudspeaker as sound;

in response to the input ambisonic audio signal and the remote audio signal, generating an output audio signal by performing acoustic echo cancellation on the input ambisonic audio signal, wherein performing the acoustic echo cancellation removes the remote audio signal from the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes introducing comfort noise into the input ambisonic audio signal;

wherein introducing the comfort noise into the input ambisonic audio signal includes estimating spatial and spectral characteristics of noise in the input ambisonic audio signal and generating comfort noise that matches the spatial and spectral characteristics of the noise in the input ambisonic audio signal;

generating an A-format encoding of the input ambisonic audio signal, wherein the A-format encoding of the input ambisonic audio signal comprises a pulse-code modulation encoding of the audio data in each channel of the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes removing the remote audio signal from the A-format encoding of the input ambisonic audio signal; and

transmitting the output audio signal from the local meeting equipment to the remote meeting equipment for playback through at least two loudspeakers in the remote meeting equipment during the bidirectional real-time communication session.

2. The method of claim 1 , further comprising:

after removing the remote audio signal from the A-format encoding of the input ambisonic audio signal, generating a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal;

wherein the output audio signal comprises the B-format encoding of the input ambisonic audio signal; and

wherein transmitting the output audio signal to the remote meeting equipment comprises transmitting the B-format encoding of the input ambisonic audio signal from the local meeting equipment to the remote meeting equipment.

3. The method of claim 1 , further comprising:

after removing the remote audio signal from the A-format encoding of the input ambisonic audio signal, generating a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal;

generating an output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal;

wherein the output audio signal comprises the output encoding of the input ambisonic audio signal; and

wherein transmitting the output audio signal to the remote meeting equipment comprises transmitting the output encoding of the input ambisonic audio signal to the remote meeting equipment.

4. The method of claim 3 wherein the remote meeting equipment includes a virtual reality headset, and further comprising:

receiving, from the virtual reality headset in the remote meeting equipment, data describing movements of the virtual reality headset;

wherein generating the output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal is responsive to the data describing movements of the virtual reality headset; and

wherein the output encoding of the input ambisonic audio signal provides three dimensional audio reflecting a current positional orientation of the virtual reality headset.

5. The method of claim 1 , further comprising:

performing automatic gain control on the input ambisonic audio signal by i) monitoring a maximum energy detected on each one of the channels in the input ambisonic audio signal, ii) determining a single adaptive gain level based on the maximum energy detected on each one of the channels in the input ambisonic audio signal, and iii) applying the single adaptive gain level to each one of the channels in the input ambisonic audio signal.

6. A method of processing ambisonic audio to provide three dimensional audio during a bidirectional real-time communication session between local meeting equipment and remote meeting equipment, comprising:

receiving, during the bidirectional real-time communication session, an input ambisonic audio signal, the input ambisonic audio signal including a plurality of channels, wherein each channel in the input ambisonic audio signal is made up of audio data representing sound captured by an ambisonic microphone located in the local meeting equipment;

receiving a remote audio signal, wherein the remote audio signal is made up of audio data representing sound captured by a microphone contained in remote meeting equipment, and wherein the remote audio signal is passed to at least one local loudspeaker located in the local meeting equipment for output by the local loudspeaker as sound;

in response to the input ambisonic audio signal and the remote audio signal, generating an output audio signal by performing acoustic echo cancellation on the input ambisonic audio signal, wherein performing the acoustic echo cancellation removes the remote audio signal from the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes introducing comfort noise into the input ambisonic audio signal;

wherein introducing the comfort noise into the input ambisonic audio signal includes estimating spatial and spectral characteristics of noise in the input ambisonic audio signal and generating comfort noise that matches the spatial and spectral characteristics of the noise in the input ambisonic audio signal;

generating an A-format encoding of the input ambisonic audio signal, wherein the A-format encoding of the input ambisonic audio signal comprises a pulse-code modulation encoding of the audio data in each channel of the input ambisonic audio signal;

generating a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes removing the remote audio signal from the B-format encoding of the input ambisonic audio signal; and

transmitting the output audio signal from the local meeting equipment to the remote meeting equipment for playback through at least two loudspeakers in the remote meeting equipment during the bidirectional real-time communication session.

7. The method of claim 6 , wherein the output audio signal comprises the B-format encoding of the input ambisonic audio signal; and

wherein transmitting the output audio signal to the remote meeting equipment comprises transmitting the B-format encoding of the input ambisonic audio signal to the remote meeting equipment.

8. The method of claim 6 , further comprising:

after removing the remote audio signal from the B-format encoding of the input ambisonic audio signal, generating an output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal;

wherein the output audio signal comprises the output encoding of the input ambisonic audio signal; and

wherein transmitting the output audio signal to the remote meeting equipment comprises transmitting the output encoding of the input ambisonic audio signal to the remote meeting equipment.

9. The method of claim 6 , further comprising:

performing automatic gain control on the input ambisonic audio signal by i) monitoring a maximum energy detected on each one of the channels in the input ambisonic audio signal, ii) determining a single adaptive gain level based on the maximum energy detected on each one of the channels in the input ambisonic audio signal, and iii) applying the single adaptive gain level to each one of the channels in the input ambisonic audio signal.

10. A method of processing ambisonic audio to provide three dimensional audio during a bidirectional real-time communication session between local meeting equipment and remote meeting equipment, comprising:

receiving, during the bidirectional real-time communication session, an input ambisonic audio signal, the input ambisonic audio signal including a plurality of channels, wherein each channel in the input ambisonic audio signal is made up of audio data representing sound captured by an ambisonic microphone located in the local meeting equipment;

receiving a remote audio signal, wherein the remote audio signal is made up of audio data representing sound captured by a microphone contained in remote meeting equipment, and wherein the remote audio signal is passed to at least one local loudspeaker located in the local meeting equipment for output by the local loudspeaker as sound;

in response to the input ambisonic audio signal and the remote audio signal, generating an output audio signal by performing acoustic echo cancellation on the input ambisonic audio signal, wherein performing the acoustic echo cancellation removes the remote audio signal from the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes introducing comfort noise into the input ambisonic audio signal;

wherein introducing the comfort noise into the input ambisonic audio signal includes estimating spatial and spectral characteristics of noise in the input ambisonic audio signal and generating comfort noise that matches the spatial and spectral characteristics of the noise in the input ambisonic audio signal;

generating an A-format encoding of the input ambisonic audio signal, wherein the A-format encoding of the input ambisonic audio signal comprises a pulse-code modulation encoding of the audio data in each channel of the input ambisonic audio signal;

generating a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal;

generating an output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes removing the remote audio signal from the output encoding of the input ambisonic audio signal; and

transmitting the output audio signal from the local meeting equipment to the remote meeting equipment for playback through at least two loudspeakers in the remote meeting equipment during the bidirectional real-time communication session.

11. The method of claim 10 , further comprising:

performing automatic gain control on the input ambisonic audio signal by i) monitoring a maximum energy detected on each one of the channels in the input ambisonic audio signal, ii) determining a single adaptive gain level based on the maximum energy detected on each one of the channels in the input ambisonic audio signal, and iii) applying the single adaptive gain level to each one of the channels in the input ambisonic audio signal.

12. An electronic system, comprising:

processing circuitry; and

memory communicably coupled to the processing circuitry, the memory storing program code for processing ambisonic audio to provide three dimensional audio during a bidirectional real-time communication session between local meeting equipment and remote meeting equipment, wherein the program code, when executed by the processing circuitry, causes the processing circuitry to:

receive, during the bidirectional real-time communication session, an input ambisonic audio signal, the input ambisonic audio signal including a plurality of channels, wherein each channel in the input ambisonic audio signal is made up of audio data representing sound captured by an ambisonic microphone located in the local meeting equipment;

receive a remote audio signal, wherein the remote audio signal is made up of audio data representing sound captured by a microphone contained in remote meeting equipment, and wherein the remote audio signal is passed to at least one local loudspeaker located in the local meeting equipment for output by the local loudspeaker as sound;

in response to the input ambisonic audio signal and the remote audio signal, generate an output audio signal by performing acoustic echo cancellation on the input ambisonic audio signal, wherein performing the acoustic echo cancellation removes the remote audio signal from the input ambisonic audio signal;

wherein the acoustic echo cancellation is performed on the input ambisonic audio signal at least in part by introducing comfort noise into the input ambisonic audio signal;

wherein the comfort noise is introduced into the input ambisonic audio signal at least in part by estimating spatial and spectral characteristics of noise in the input ambisonic audio signal and generating comfort noise that matches the spatial and spectral characteristics of the noise in the input ambisonic audio signal;

generate an A-format encoding of the input ambisonic audio signal, wherein the A-format encoding of the input ambisonic audio signal comprises a pulse-code modulation encoding of the audio data in each channel of the input ambisonic audio signal;

wherein the acoustic echo cancellation is performed on the input ambisonic audio signal at least in part by removing the remote audio signal from the A-format encoding of the input ambisonic audio signal; and

transmit the output audio signal from the local meeting equipment to the remote meeting equipment for playback through at least two loudspeakers in the remote meeting equipment during the bidirectional real-time communication session.

13. The electronic system of claim 12 , wherein the program code, when executed by the processing circuitry, further causes the processing circuitry to:

after removing the remote audio signal from the A-format encoding of the input ambisonic audio signal, generate a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal, and wherein the output audio signal comprises the B-format encoding of the input ambisonic audio signal; and

transmit the output audio signal to the remote meeting equipment by transmitting the B-format encoding of the input ambisonic audio signal from the local meeting equipment to the remote meeting equipment.

14. The electronic system of claim 12 , wherein the program code, when executed by the processing circuitry, further causes the processing circuitry to:

after removing the remote audio signal from the A-format encoding of the input ambisonic audio signal, generate a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal;

generate an output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal, wherein the output audio signal comprises the output encoding of the input ambisonic audio signal; and

transmit the output audio signal to the remote meeting equipment comprises by transmitting the output encoding of the input ambisonic audio signal to the remote meeting equipment.

15. The electronic system of claim 14 , wherein the remote meeting equipment includes a virtual reality headset, and wherein the program code, when executed by the processing circuitry, further causes the processing circuitry to:

receive, from the virtual reality headset in the remote meeting equipment, data describing movements of the virtual reality headset;

generate the output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal responsive to the data describing movements of the virtual reality headset; and

wherein the output encoding of the input ambisonic audio signal provides three dimensional audio reflecting a current positional orientation of the virtual reality headset.

16. The electronic system of claim 12 , wherein the program code, when executed by the processing circuitry, further causes the processing circuitry to:

perform automatic gain control on the input ambisonic audio signal by i) monitoring a maximum energy detected on each one of the channels in the input ambisonic audio signal, ii) determining a single adaptive gain level based on the maximum energy detected on each one of the channels in the input ambisonic audio signal, and iii) applying the single adaptive gain level to each one of the channels in the input ambisonic audio signal.

17. An electronic system, comprising:

processing circuitry; and

memory communicably coupled to the processing circuitry, the memory storing program code for processing ambisonic audio to provide three dimensional audio during a bidirectional real-time communication session between local meeting equipment and remote meeting equipment, wherein the program code, when executed by the processing circuitry, causes the processing circuitry to:

receive, during the bidirectional real-time communication session, an input ambisonic audio signal, the input ambisonic audio signal including a plurality of channels, wherein each channel in the input ambisonic audio signal is made up of audio data representing sound captured by an ambisonic microphone located in the local meeting equipment;

receive a remote audio signal, wherein the remote audio signal is made up of audio data representing sound captured by a microphone contained in remote meeting equipment, and wherein the remote audio signal is passed to at least one local loudspeaker located in the local meeting equipment for output by the local loudspeaker as sound;

in response to the input ambisonic audio signal and the remote audio signal, generate an output audio signal by performing acoustic echo cancellation on the input ambisonic audio signal, wherein performing the acoustic echo cancellation removes the remote audio signal from the input ambisonic audio signal;

wherein the acoustic echo cancellation is performed on the input ambisonic audio signal at least in part by introducing comfort noise into the input ambisonic audio signal;

wherein the comfort noise is introduced into the input ambisonic audio signal at least in part by estimating spatial and spectral characteristics of noise in the input ambisonic audio signal and generating comfort noise that matches the spatial and spectral characteristics of the noise in the input ambisonic audio signal;

generate an A-format encoding of the input ambisonic audio signal, wherein the A-format encoding of the input ambisonic audio signal comprises a pulse-code modulation encoding of the audio data in each channel of the input ambisonic audio signal;

generate a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal;

wherein the acoustic echo cancellation is performed on the input ambisonic audio signal at least in part by removing the remote audio signal from the B-format encoding of the input ambisonic audio signal; and

transmit the output audio signal from the local meeting equipment to the remote meeting equipment for playback through at least two loudspeakers in the remote meeting equipment during the bidirectional real-time communication session.

18. The electronic system of claim 17 , wherein the output audio signal comprises the B-format encoding of the input ambisonic audio signal; and

wherein the program code, when executed by the processing circuitry, further causes the processing circuitry to transmit the output audio signal to the remote meeting equipment by transmitting the B-format encoding of the input ambisonic audio signal to the remote meeting equipment.

19. The electronic system of claim 17 , wherein the program code, when executed by the processing circuitry, further causes the processing circuitry to:

after removing the remote audio signal from the B-format encoding of the input ambisonic audio signal, generate an output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal;

wherein the output audio signal comprises the output encoding of the input ambisonic audio signal; and

wherein the output audio signal is transmitted to the remote meeting equipment at least in part by transmitting the output encoding of the input ambisonic audio signal to the remote meeting equipment.

20. An electronic system, comprising:

processing circuitry; and

memory communicably coupled to the processing circuitry, the memory storing program code for processing ambisonic audio to provide three dimensional audio during a bidirectional real-time communication session between local meeting equipment and remote meeting equipment, wherein the program code, when executed by the processing circuitry, causes the processing circuitry to:

receive, during the bidirectional real-time communication session, an input ambisonic audio signal, the input ambisonic audio signal including a plurality of channels, wherein each channel in the input ambisonic audio signal is made up of audio data representing sound captured by an ambisonic microphone located in the local meeting equipment;

receive a remote audio signal, wherein the remote audio signal is made up of audio data representing sound captured by a microphone contained in remote meeting equipment, and wherein the remote audio signal is passed to at least one local loudspeaker located in the local meeting equipment for output by the local loudspeaker as sound;

in response to the input ambisonic audio signal and the remote audio signal, generate an output audio signal by performing acoustic echo cancellation on the input ambisonic audio signal, wherein performing the acoustic echo cancellation removes the remote audio signal from the input ambisonic audio signal;

wherein the acoustic echo cancellation is performed on the input ambisonic audio signal at least in part by introducing comfort noise into the input ambisonic audio signal;

wherein the comfort noise is introduced into the input ambisonic audio signal at least in part by estimating spatial and spectral characteristics of noise in the input ambisonic audio signal and generating comfort noise that matches the spatial and spectral characteristics of the noise in the input ambisonic audio signal;

generate an A-format encoding of the input ambisonic audio signal, wherein the A-format encoding of the input ambisonic audio signal comprises a pulse-code modulation encoding of the audio data in each channel of the input ambisonic audio signal;

generate a B-format encoding of the input ambisonic audio signal from the A-format encoding of the input ambisonic audio signal, wherein the B-format encoding of the input audio signal is a loudspeaker configuration independent encoding of the input ambisonic audio signal;

generate an output encoding of the input ambisonic audio signal from the B-format encoding of the input ambisonic audio signal;

wherein the acoustic echo cancellation is performed on the input ambisonic audio signal at least in part by removing the remote audio signal from the output encoding of the input ambisonic audio signal; and

transmit the output audio signal from the local meeting equipment to the remote meeting equipment for playback through at least two loudspeakers in the remote meeting equipment during the bidirectional real-time communication session.

21. A computer program product, comprising:

a non-transitory computer readable medium storing program code for processing ambisonic audio to provide three dimensional audio during a bidirectional real-time communication session between local meeting equipment and remote meeting equipment, the set of instructions, when carried out by at least one processor, causing the processor to perform a method of:

receiving, during the bidirectional real-time communication session, an input ambisonic audio signal, the input ambisonic audio signal including a plurality of channels, wherein each channel in the input ambisonic audio signal is made up of audio data representing sound captured by an ambisonic microphone located in the local meeting equipment;

receiving a remote audio signal, wherein the remote audio signal is made up of audio data representing sound captured by a microphone contained in remote meeting equipment, and wherein the remote audio signal is passed to at least one local loudspeaker located in the local meeting equipment for output by the local loudspeaker as sound;

in response to the input ambisonic audio signal and the remote audio signal, generating an output audio signal by performing acoustic echo cancellation on the input ambisonic audio signal, wherein performing the acoustic echo cancellation removes the remote audio signal from the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes introducing comfort noise into the input ambisonic audio signal;

wherein introducing the comfort noise into the input ambisonic audio signal includes estimating spatial and spectral characteristics of noise in the input ambisonic audio signal and generating comfort noise that matches the spatial and spectral characteristics of the noise in the input ambisonic audio signal;

generating an A-format encoding of the input ambisonic audio signal, wherein the A-format encoding of the input ambisonic audio signal comprises a pulse-code modulation encoding of the audio data in each channel of the input ambisonic audio signal;

wherein performing the acoustic echo cancellation on the input ambisonic audio signal includes removing the remote audio signal from the A-format encoding of the input ambisonic audio signal; and

transmitting the output audio signal from the local meeting equipment to the remote meeting equipment for playback through at least two loudspeakers in the remote meeting equipment during the bidirectional real-time communication session.

Assignments (11)