Separating and rendering voice and ambience signals by offsetting impact of device movements
Processing of ambience and speech can include extracting from audio signals, ambience and speech signals. One or more spatial parameters can be generated that define spatial characteristics of ambience sound in the one or more ambience audio signals. The primary speech signal, the one or more ambience audio signals, and the spatial parameters can be encoded into one or more encoded data streams. Other aspects are described and claimed.
1. A method performed by a processor of a device having a plurality of microphones, comprising:
receiving a plurality of audio signals from the plurality of microphones, the plurality of microphones capturing a sound field;
processing the audio signals into a plurality of frequency domain signals;
extracting, from the frequency domain signals, a primary speech signal;
extracting, from the frequency domain signals, one or more ambience audio signals;
generating one or more spatial parameters defining spatial characteristics of an ambience sound in the one or more ambience audio signals, the one or more spatial parameters include a location of an ambience sound source;
detecting a location or an orientation of the device, as tracking data;
modifying the one or more spatial parameters based on the tracking data by offsetting a relative movement of the ambience sound source, the relative movement caused by a change in the location or orientation of the device, to maintain a location of the ambience sound source constant during playback; and
encoding the primary speech signal, the one or more ambience audio signals, and the as modified spatial parameters into one or more encoded data streams.
2. The method of claim 1 , wherein the tracking data is generated based on one or more sensors, the sensors including one or more of the following: a camera, a set of microphones, a gyroscope, an accelerometer, and a GPS receiver.
3. The method of claim 1 , wherein the tracking data is generated based on images captured by a camera, including comparing a first image with a second image and determining a change in the location or orientation of the device based on the comparison.
4. The method of claim 1 , wherein the tracking data is generated based on estimation of sound source locations in the sound field, and detected changes in the sound source locations in the sound field, indicating a change in the location or orientation of the device.
5. The method of claim 1 , further comprising encoding the tracking data in the encoded data streams.
6. The method of claim 1 , wherein the primary speech signal is encoded without corresponding spatial parameters and is to be played back by a playback device without spatialization.
7. The method of claim 1 , wherein the primary speech signal is identified as being the primary speech signal based on a detected location of a speaker relative to the device.
8. The method of claim 1 , further comprising transmitting the encoded data streams in real-time, to a playback device, wherein the encoded data streams further includes a stream of images in sync with the primary speech signal and the ambience audio signals.
9. A method performed by a playback device, for playback of sound captured by a capture device, comprising:
receiving from the capture device one or more encoded data streams;
decoding the one or more encoded data streams to extract a primary speech signal, one or more ambience audio signals representing an ambience sound source, and one or more spatial parameters of the one or more ambience audio signals wherein the one or more spatial parameters include a location of the ambience sound source;
modifying the one or more spatial parameters of the one or more ambience audio signals based on tracking data received in and decoded from the one or more encoded data streams, the tracking data including a location or orientation of the capture device, by offsetting a relative movement of the ambience sound source caused by a change in the location or orientation of the capture device, to maintain a location of the ambience sound source constant during playback in the playback device;
determining, based on the one or more spatial parameters, one or more impulse responses;
convolving each of the one or more ambience audio signals with the one or more impulse responses resulting in spatialized ambience audio signals;
processing the spatialized ambience audio signals and the primary speech signal to produce a plurality of time domain channel signals; and
driving a plurality of speakers based on the plurality of time domain channel signals.
10. The method of claim 9 , further comprising defining or modifying a playback level of the one or more ambience audio signals based on a user input.
11. The method of claim 10 , wherein the user input is received through a graphical user interface of the playback device.
12. The method of claim 9 , wherein the tracking data is generated based on one or more sensors of the capture device, the sensors including one or more of the following: a camera, a set of microphones, a gyroscope, an accelerometer, and a GPS receiver.
13. The method of claim 9 , further comprising defining or modifying a playback level of the one or more ambience audio signals based on a) a speech to noise ratio, b) a content type, or c) a detected noise in a playback environment.
14. The method of claim 9 , wherein the primary speech signal is played directly through the plurality of speakers without spatialization.
15. An article of manufacture comprising: a non-transitory machine readable medium having stored therein instructions that, when executed by a processor of an audio capture device, cause the audio capture device to perform the following:
receiving a plurality of audio signals from a plurality of microphones that capture a sound field;
processing the audio signals into a plurality of frequency domain signals;
extracting, from the frequency domain signals, a primary speech signal;
extracting, from the frequency domain signals, one or more ambience audio signals;
generating one or more spatial parameters defining spatial characteristics of ambience in the one or more ambience audio signals, wherein the one or more spatial parameters include a location of an ambience sound source;
detecting a location or an orientation of the audio capture device, as tracking data;
modifying the one or more spatial parameters based on the tracking data by offsetting a relative movement of the ambience sound source caused by a change in the location or orientation of the audio capture device, to maintain a location of the ambience sound source constant during playback; and
encoding the primary speech signal, the ambience audio signals, and the spatial parameters into one or more encoded data streams.
16. An article of manufacture comprising: a non-transitory machine readable medium having stored therein instructions that, when executed by a processor of a playback device, cause the playback device to perform the following:
receiving from a capture device one or more encoded data streams;
decoding the one or more encoded data streams to extract a primary speech signal, one or more ambience audio signals representing an ambience sound source, and one or more spatial parameters that describe a spatial characteristic of the one or more ambience audio signals including a location of the ambience sound source;
modifying the spatial parameters of the one or more ambience audio signals based on tracking data received in and decoded from the encoded data streams, the tracking data including a location or orientation of the capture device, by offsetting a relative movement of the ambience sound source caused by a change in the location or orientation of the capture device, to maintain a location of the ambience sound source constant during playback;
determining, based on the one or more spatial parameters, one or more impulse responses;
convolving each of the one or more ambience audio signals with the one or more impulse responses resulting in spatialized ambience audio signals;
processing the spatialized ambience audio signals and the primary speech signal to produce a plurality of time domain channel signals; and
driving a plurality of speakers based on the plurality of time domain channel signals.