Audio encoding with compressed ambience
An audio device can sense sound in a physical environment using a plurality of microphones to generate a plurality of microphone signals. Clean speech can be extracted from microphone signals. Ambience can be extracted from the microphone signals. The clean speech can be encoded at a first compression level. The ambience can be encoded at a second compression level that is higher than the first compression level. Other aspects are also described and claimed.
1. A method performed by an audio device, comprising:
receiving, in a bit stream, a) an encoded speech signal containing speech sensed by a plurality of microphones in a physical environment, the encoded speech signal compressed to have a first bit rate, b) an encoded ambient signal containing ambient sound sensed by the plurality of microphones in the physical environment, the encoded ambient signal is compressed to have a second bit rate that is lower than the first bit rate; and c) one or more acoustic parameters of the physical environment;
decoding the encoded speech signal and the encoded ambient signal; and
applying the one or more acoustic parameters to a decoded speech signal for playback through a plurality of speakers.
2. The method of claim 1 , wherein the one or more acoustic parameters includes one or more binaural room impulse responses (BRIRs).
3. The method of claim 2 , wherein the BRIRs are applied to the decoded speech signal to spatialize the speech for playback through a left headphone speaker and a right headphone speaker of the plurality of speakers.
4. The method of claim 1 , wherein the one or more acoustic parameters includes a reverberation time or a pattern of early reflections of the physical environment.
5. The method of claim 1 , wherein applying the one or more acoustic parameters to the decoded speech signal generates a speech signal with a reverberant component for playback through the plurality of speakers.
6. The method of claim 1 , wherein the audio device includes a plurality of microphones that are integral to the audio device; the audio device being one or more of: a head-worn device, a mobile device with a display, a smart speaker, or a virtual reality headset; and the bit stream is received from another audio device through a communication protocol.
7. The method of claim 6 , wherein the audio device has a wireless transmitter, and the communication protocol is a wireless communication protocol.
8. The method of claim 1 , wherein the one or more acoustic parameters includes a reverberation decay time or a pattern of early reflections of the physical environment.
9. The method of claim 1 , wherein the one or more acoustic parameters includes one or more impulse responses of the physical environment.
10. A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations, comprising:
receiving, in a bit stream, a) an encoded speech signal containing speech sensed by a plurality of microphones in a physical environment, the encoded speech signal is compressed to have a first bit rate, b) an encoded ambient signal containing ambient sound sensed by the plurality of microphones in the physical environment, the encoded ambient signal is compressed to have a second bit rate that is lower than the first bit rate; and c) one or more acoustic parameters of the physical environment;
decoding the encoded speech signal and the encoded ambient signal; and
applying the one or more acoustic parameters to a decoded speech signal for playback through a plurality of speakers.
11. The non-transitory computer readable medium storing instructions of claim 10 , the operations further comprising:
decoding, from the bit stream, one or more spatial parameters associated with a) the ambient sound, or b) the speech, the one or more spatial parameters defining spatial locations of the ambient sound or the speech in the physical environment; and
applying the one or more spatial parameters to a decoded ambient signal or the decoded speech signal.
12. The non-transitory computer readable medium storing instructions of claim 10 , wherein a bit rate of the encoded speech signal is 96 KB/sec or greater.
13. The non-transitory computer readable medium storing instructions of claim 10 , wherein a bit rate of the encoded ambient signal is less than one tenth of a bit rate of the encoded speech signal.
14. The non-transitory computer readable medium storing instructions of claim 10 , wherein the encoded speech signal does not contain reverberant or ambient sound components.
15. The non-transitory computer readable medium storing instructions of claim 10 , the operations further comprising:
rendering a video stream onto a display of an audio device, the video stream including an avatar or real-life depiction of a speaker and the physical environment.
16. An audio device, comprising: a plurality of microphones that form a microphone array that generate a plurality of microphone signals representing sound sensed in a physical environment; and one or more processors configured to: extract clean speech from the plurality of microphone signals; extract ambience from the plurality of microphone signals; determine, based on the plurality of microphone signals, one or more acoustic parameters of the physical environment, wherein the one or more acoustic parameters include one or more of: a reverberation time, a pattern of early reflections, or one or more impulse responses of the physical environment; and encode, in a bit stream a) the clean speech by compressing the clean speech into an encoded speech signal at a first bit rate,) the ambience by compressing the ambience into an encoded ambience signal at a second bit rate that is lower than the first bit rate, and c) the one or more acoustic parameters of the physical environment, the one or more acoustic parameters encoded to be applied to the clean speech by a receiving device.
17. The audio device of claim 16 , wherein the plurality of microphones is integral to the audio device being a head-worn device, a mobile device with display, a smart speaker, or a virtual reality headset, and wherein the audio device is to transmit the bit stream to a second device through a wireless communication protocol.
18. The audio device of claim 16 , wherein the clean speech does not contain reverberant or ambient sound components.
19. The audio device of claim 16 , wherein the one or more impulse responses includes a binaural room impulse response (BRIR).
20. The audio device of claim 16 , wherein the one or more acoustic parameters are determined based on a) one or more images of the physical environment, and b) measured reverberation of the physical environment based on the plurality of microphone signals.