Audio encoding and decoding to generate binaural virtual spatial signals
An audio encoder comprises a multi-channel receiver ( 401 ) which receives an M-channel audio signal where M>2. A down-mix processor ( 403 ) down-mixes the M-channel audio signal to a first stereo signal and associated parametric data and a spatial processor ( 407 ) modifies the first stereo signal to generate a second stereo signal in response to the associated parametric data and spatial parameter data for a binaural perceptual transfer function, such as a Head Related Transfer Function (HRTF). The second stereo signal is a binaural signal and may specifically be a (3D) virtual spatial signal. An output data stream comprising the encoded data and the associated parametric data is generated by an encode processor ( 411 ) and an output processor ( 413 ). The HRTF processing may allow the generation of a (3D) virtual spatial signal by conventional stereo decoders. A multi-channel decoder may reverse the process of the spatial processor ( 407 ) to generate an improved quality multi-channel signal.
1. An audio encoder comprising:
a receiver for receiving an M-channel audio signal where M>2;
a down-mixing processor for down-mixing the M-channel audio signal to provide a first stereo signal and associated parametric data;
a modifying processor for modifying sub band values of the first stereo signal by multiplying the sub band values of the first stereo signal with sub band dependent matrix values to generate sub band values of a second stereo signal, wherein the sub band dependent matrix values are based on the associated parametric data and first spatial parameter data of a binaural perceptual transfer function, the second stereo signal being a binaural signal;
an encode processor for encoding the second stereo signal to generate encoded data; and
an output processor for generating an output data stream comprising the encoded data and the associated parametric data.
2. The encoder of claim 1 wherein the modifying processor is arranged to generate the second stereo signal by calculating the sub band values of the second stereo signal based on: the associated parametric data, the first spatial parameter data, and the sub band values of the first stereo signal.
3. The encoder of claim 2 wherein the modifying processor is arranged to generate sub band values for a first sub band of the second stereo signal based on a multiplication of corresponding stereo sub band values for the first stereo signal by a first sub band matrix; the modifying processor being configured for determining data values of the first sub band matrix based on: the associated parametric data, and the first spatial parameter data for the first sub band.
4. The encoder of claim 3 wherein the modifying processor is configured for converting a data value of at least one of: the first stereo signal, the associated parametric data, and sub band spatial parameter data associated with a sub band having a frequency interval different from the first sub band interval, to provide a corresponding data value for the first sub band.
5. The encoder of claim 3 wherein the modifying processor is arranged to determine the stereo sub band values L B , R B for the first sub band of the second stereo signal substantially as:
[
L
B
R
B
]
=
[
h
11
h
12
h
21
h
22
]
[
L
0
R
0
]
,
wherein L O , R O are corresponding sub band values of the first stereo signal and the modifying processor is configured to determine data values of the multiplication matrix substantially as:
h 11 =m 11 H L ( L )+ m 21 H L ( R )+ m 31 H L ( C )
h 12 =m 12 H L ( L )+ m 22 H L ( R )+ m 32 H L ( C )
h 21 =m 11 H R ( L )+ m 21 H R ( R )+ m 31 H R ( C )
h 22 =m 12 H R ( L )+ m 22 H R ( R )+ m 32 H R ( C )
where m k,l are parameters determined based on associated parametric data for a down-mix by the down-mixing processor of channels L, R and C to provide the first stereo signal; and H J (X) is determined based on channel X spatial parameter data for channel X to provide output channel J of the second stereo signal.
6. The encoder of claim 5 wherein at least one of channels L and R correspond to a down-mix of at least two down-mixed channels, and the modifying processor is configured to determine H J (X) based on a weighted combination of down-mixed channel spatial parameter data for the at least two down-mixed channels.
7. The encoder of claim 6 wherein the modifying processor is configured to determine a weighting of down-mixed channel spatial parameter data for the at least two down-mixed channels based on a relative energy measure for the at least two down-mixed channels.
8. The encoder of claim 1 wherein the first spatial parameter data includes at least one parameter selected from the group consisting of:
an average level per sub band parameter;
an average arrival time parameter;
a phase of at least one stereo channel;
a timing parameter;
a group delay parameter;
a phase between stereo channels; or
a cross channel correlation parameter.
9. The encoder of claim 1 wherein the output processor is arranged to include sound source position data in the output stream.
10. The encoder of claim 1 wherein the output processor is arranged to include at least some of the first spatial parameter data in the output stream.
11. The encoder of claim 1 comprising means for determining the first spatial parameter data based on desired sound signal positions.
12. An audio decoder comprising:
an input receiver for receiving input data comprising a first stereo signal and parametric data associated with a down-mixed second stereo signal of an M-channel audio signal where M>2, the first stereo signal being a binaural signal corresponding to the M-channel audio signal;
a modifying processor for modifying sub band values of the first stereo signal by multiplying the sub band values of the first stereo signal with sub band dependent inverse matrix values to generate sub band values of the down-mixed second stereo signal, wherein the sub band dependent inverse matrix values are based on the parametric data and first spatial parameter data of a binaural perceptual transfer function, the first spatial parameter data being associated with the first stereo signal.
13. The decoder of claim 12 further comprising a multi-channel decoder for generating the M-channel audio signal based on the down-mixed second stereo signal and the parametric data.
14. The decoder of claim 12 wherein the modifying processor is arranged to generate the down-mixed second stereo signal by calculating the sub band values of the down-mixed second stereo signal based on: the associated parametric data, the first spatial parameter data, and the sub band values of the first stereo signal.
15. The decoder of claim 14 wherein the modifying processor is configured to generate sub band values for a first sub band of the down-mixed second stereo signal depending on a multiplication of corresponding stereo sub band values for the first stereo signal by a first sub band matrix; the modifying processor being configured for determining data values of the first sub band matrix based on parametric data and binaural perceptual transfer function parameter data for the first sub band.
16. The decoder of claim 12 wherein the input data comprises at least some of the first spatial parameter data.
17. The decoder of claim 12 wherein the input data comprises sound source position data and the decoder comprises a parameter processor for determining the first spatial parameter data based on the sound source position data.
18. The decoder of claim 12 further comprising:
a spatial decoder unit for producing a pair of binaural output channels by modifying the first stereo signal based on the associated parametric data and second spatial parameter data for a second binaural perceptual transfer function, the second spatial parameter data being different than the first spatial parameter data.
19. The decoder of claim 18 wherein the spatial decoder unit comprises:
a parameter conversion unit for converting the parametric data into binaural synthesis parameters using the second spatial parameter data, and
a spatial synthesis unit for synthesizing the pair of binaural channels using the binaural synthesis parameters and the first stereo signal.
20. The decoder of claim 19 wherein the binaural synthesis parameters comprise matrix coefficients for a 2 by 2 matrix relating stereo samples of the down-mixed stereo signal to stereo samples of the pair of binaural output channels.
21. The decoder of claim 19 wherein the binaural synthesis parameters comprise matrix coefficients for a 2 by 2 matrix relating stereo sub band samples of the first stereo signal to stereo samples of the pair of binaural output channels.
22. A method of audio encoding, the method comprising:
receiving an M-channel audio signal where M>2;
down-mixing the M-channel audio signal to provide a first stereo signal and associated parametric data;
modifying sub band values of the first stereo signal by multiplying the sub band values of the first stereo signal with sub band dependent matrix values to generate sub band values of a second stereo signal, wherein the sub band dependent matrix values are based on the associated parametric data and first spatial parameter data of a binaural perceptual transfer function, the second stereo signal being a binaural signal;
encoding the second stereo signal to generate encoded data; and
generating an output data stream comprising the encoded data and the associated parametric data.
23. A method of audio decoding, the method comprising:
receiving input data comprising a first stereo signal and parametric data associated with a down-mixed stereo signal of an M-channel audio signal where M>2, the first stereo signal being a binaural signal corresponding to the M-channel audio signal; and
modifying sub band values of the first stereo signal by multiplying the sub band values of the first stereo signal with sub band dependent inverse matrix values to generate sub band values of the down-mixed stereo signal, wherein the sub band dependent inverse matrix values are based on: the parametric data, and first spatial parameter data of a binaural perceptual transfer function, the first spatial parameter data being associated with the first stereo signal.
24. A non-transitory computer readable storage medium encoded with instructions for controlling a processor for performing a method of audio encoding, the method comprising:
receiving an M-channel audio signal where M>2;
down-mixing the M-channel audio signal to provide a first stereo signal and associated parametric data;
modifying sub band values of the first stereo signal by multiplying the sub band values of the first stereo signal with sub band dependent matrix values to generate sub band values of a second stereo signal, wherein the sub band dependent matrix values are based on the associated parametric data and first spatial parameter data of a binaural perceptual transfer function, the second stereo signal being a binaural signal;
encoding the second stereo signal to generate encoded data; and
generating an output data stream comprising the encoded data and the associated parametric data.
25. An audio recording device comprising an encoder according to claim 1 .
26. An audio playing device comprising a decoder according to claim 12 .
27. A non-transitory computer readable storage medium encoded with instructions for controlling a processor for performing a method of audio decoding, the method comprising:
receiving input data comprising a first stereo signal and instructions comprising control data for controlling the audio decoding of the first stereo signal, the control data including parametric data associated with a down-mixed second stereo signal of an M-channel audio signal where M>2, the first stereo signal being a binaural signal corresponding to the M-channel audio signal; and
modifying sub band values of the first stereo signal by multiplying the sub band values of the first stereo signal with sub band dependent inverse matrix values to generate sub band values of the down-mixed second stereo signal, wherein the sub band dependent inverse matrix values are based on the parametric data and first spatial parameter data of a binaural perceptual transfer function, the first spatial parameter data being associated with the first stereo signal.