Audio encoding device and method
A method and a device encode N audio signals, from N microphones where N≥3. For each pair of the N audio signals an angle of incidence of direct sound is estimated. A-format direct sound signals are derived from the estimated angles of incidence by deriving from each estimated angle an A-format direct sound signal. Each A-format direct sound signal is a first-order virtual microphone signal, for example, a cardioids signal.
1. An audio encoding device, for encoding N audio signals, from N microphones where N≥3, the audio encoding device comprising:
a delay estimator configured to estimate angles of incidence of direct sound by estimating, for each pair of the N audio signals, an angle of incidence of the direct sound, and a beam deriver configured to derive A-format direct sound signals from the estimated angles of incidence by deriving, from each of the estimated angles of incidence, a respective one of the A-format direct sound signals, each of the A-format direct sound signals being a first-order virtual microphone signal; and
an encoder configured to encode the A-format direct sound signals in first-order ambisonic B-format direct sound signals by applying a transformation matrix to the A-format direct sound signals,
wherein N=3,
wherein the audio encoding device comprises a short time Fourier transformer configured to perform a short time Fourier transformation on each of the N audio signals x 1 , x 2 , x 3 , resulting in N short time Fourier transformed audio signals X 1 [k,i], X 2 [k,i], X 3 [k,i],
wherein the delay estimator is configured to:
determine cross spectra of each pair of the short time Fourier transformed audio signals according to:
X 12 [ k,i ]=α X X 1 [ k,i ] X* 2 [ k,i ]+(1−α X ) X 12 [ k− 1, i ],
X 13 [ k,i ]=α X X 1 [ k,i ] X* 3 [ k,i ]+(1−α X ) X 13 [ k− 1, i ], and
X 23 [ k,i ]=α X X 2 [ k,i ] X* 3 [ k,i ]+(1−α X ) X 23 [ k− 1, i ],
determine an angle of the complex cross spectrum of each pair of the short time Fourier transformed audio signals according to:
ψ
~
12
[
k
,
i
]
=
arctan
j
X
12
[
k
,
i
]
X
12
*
[
k
,
i
]
X
12
[
k
,
i
]
+
X
12
*
[
k
,
i
]
,
ψ
~
13
[
k
,
i
]
=
arctan
j
X
13
[
k
,
i
]
X
13
*
[
k
,
i
]
X
13
[
k
,
i
]
+
X
13
*
[
k
,
i
]
,
and
ψ
~
23
[
k
,
i
]
=
arctan
j
X
23
[
k
,
i
]
X
23
*
[
k
,
i
]
X
23
[
k
,
i
]
+
X
23
*
[
k
,
i
]
,
perform a phase unwrapping to {tilde over (ψ)} 12 , {tilde over (ψ)} 13 , {tilde over (ψ)} 23 , resulting in ψ 12 , ψ 13 , ψ 23 , estimate the delay in number of samples according to:
δ 12 [ k,i ]=( N STFT /2+1)/( i π)ψ 12 [ k,i ],
δ 13 [ k,i ]=( N STFT /2+1)/( i π)ψ 13 [ k,i ], and
δ 23 [ k,i ]=( N STFT /2+1)/( i π)ψ 23 [ k,i ], if i≤i alias
or
δ 12 [ k,i ]=( N STFT /2+1)/( i π)Ψ 12 [ k,i ],
δ 13 [ k,i ]=( N STFT /2+1)/( i π)Ψ 13 [ k,i ], and
δ 23 [ k,i ]=( N STFT /2+1)/( i π)Ψ 23 [ k,i ], if i>i alias
estimate the delay in seconds according to:
τ
12
[
k
,
i
]
=
δ
12
[
k
,
i
]
f
s
,
τ
13
[
k
,
i
]
=
δ
13
[
k
,
i
]
f
s
,
and
τ
23
[
k
,
i
]
=
δ
23
[
k
,
i
]
f
s
,
and estimate the angles of incidence according to:
θ
12
[
k
,
i
]
=
arcsin
(
c
τ
12
[
k
,
i
]
d
mic
)
,
θ
13
[
k
,
i
]
=
arcsin
(
c
τ
13
[
k
,
i
]
d
mic
)
,
and
θ
23
[
k
,
i
]
=
arcsin
(
c
τ
23
[
k
,
i
]
d
mic
)
,
and
wherein:
x 1 is a first audio signal of the N audio signals,
x 2 is a second audio signal of the N audio signals,
x 3 is a third audio signal of the N audio signals,
X 1 is a first short time Fourier transformed audio signal of the short time Fourier transformed audio signals,
X 2 is a second short time Fourier transformed audio signal of the short time Fourier transformed audio signals,
X 3 is a third short time Fourier transformed audio signal of the short time Fourer transformed audio signals,
k is a frame of the short time Fourier transformed audio signals, and
i is a frequency bin of the short time Fourier transformed audio signals,
X 12 is a cross spectrum of a pair of X 1 and X 2 ,
X 13 is a cross spectrum of a pair of X 1 and X 3 ,
X 23 is a cross spectrum of a pair of X 2 and X 3 ,
α X is a forgetting factor,
X* is a conjugate complex of X,
j is the imaginary unit,
{tilde over (ψ)} 12 is an angle of the complex cross spectrum of X 12 ,
{tilde over (ψ)} 13 is an angle of the complex cross spectrum of X 13 ,
{tilde over (ψ)} 23 is an angle of the complex cross spectrum of X 23 ,
i alias is a frequency bin corresponding to an aliasing frequency,
f s is a sampling frequency,
d mic is a distance of the microphones, and
c is the speed of sound.
2. The audio encoding device according to claim 1 ,
wherein the beam deriver is configured to:
determine cardioid directional responses according to:
D
12
[
k
,
i
]
=
1
2
(
1
+
cos
(
θ
12
[
k
,
i
]
-
π
2
)
)
,
D
13
[
k
,
i
]
=
1
2
(
1
+
cos
(
θ
13
[
k
,
i
]
-
π
2
)
)
,
and
D
13
[
k
,
i
]
=
1
2
(
1
+
cos
(
θ
23
[
k
,
i
]
-
π
2
)
)
,
and
derive the A-format direct sound signals according to:
A 12 [ k,i ]= D 12 [ k,i ] X 1 [ k,i ],
A 13 [ k,i ]= D 13 [ k,i ] X 1 [ k,i ], and
A 23 [ k,i ]= D 23 [ k,i ] X 1 [ k,i ],
wherein:
D is a cardioid directional response, and
A is an A-format direct sound signal of the A-format direct sound signals.
3. The audio encoding device according to claim 2 ,
wherein the encoder is configured to encode the A-format direct sound signals to the first-order ambisonic B-format direct sound signals according to:
[
R
W
R
X
R
Y
]
=
Γ
-
1
[
A
12
A
13
A
23
]
,
wherein:
R w is a first, zero-order ambisonic B-format direct sound signal,
R x is a first, first-order ambisonic B-format direct sound signal among the first-order ambisonic B-format direct sound signals,
R y is a second, first-order ambisonic B-format direct sound signal among the first-order ambisonic B-format direct sound signals, and
Γ −1 is the transformation matrix.
4. The audio encoding device according to claim 1 , comprising
a direction of arrival estimator configured to estimate a direction of arrival from the first-order ambisonic B-format direct sound signals, and
a higher order ambisonic encoder configured to encode higher order ambisonic B-format direct sound signals using the first-order ambisonic B-format direct sound signals and the estimated direction of arrival, wherein higher order ambisonic B-format direct sound signals have an order higher than one.
5. The audio encoding device according to claim 4 ,
wherein the direction of arrival estimator is configured to estimate the direction of arrival according to:
θ
XY
[
k
,
i
]
=
arctan
R
Y
[
k
,
i
]
R
X
[
k
,
i
]
,
and
wherein θ XY [k,i] is the direction of arrival of the direct sound of frame k and frequency bin i.
6. The audio encoding device according to claim 5 ,
wherein the higher order ambisonic B-format direct sound signals comprise second order ambisonic B-format direct sound signals limited to two dimensions,
wherein the higher order ambisonic encoder is configured to encode the second order ambisonic B-format direct sound signals according to:
R
R
Δ
=
(
3
sin
2
ϕ
-
1
)
/
2
=
-
1
/
2
,
R
S
Δ
=
3
/
2
cos
θsin
2
ϕ
=
0
,
R
T
Δ
=
3
/
2
sin
θ
sin
2
ϕ
=
0
,
R
U
Δ
=
3
/
2
cos
2
θ
cos
2
ϕ
=
3
/
2
cos
2
θ
X
Y
,
and
R
V
Δ
=
3
/
2
sin
2
θ
cos
2
ϕ
=
3
/
2
sin
2
θ
X
Y
,
and
wherein:
R R is a first, second-order ambisonic B-format direct sound signal among the second order ambisonic B-format direct signals,
R S is a second, second-order ambisonic B-format direct sound signal among the second order ambisonic B-format direct signals,
R T is a third, second-order ambisonic B-format direct sound signal among the second order ambisonic B-format direct signals,
R U is a fourth, second-order ambisonic B-format direct sound signal among the second order ambisonic B-format direct signals,
R V is a fifth, second-order ambisonic B-format direct sound signal among the second order ambisonic B-format direct signals,
denotes “defined as”,
Φ is an elevation angle, and
θ is an azimuth angle.
7. The audio encoding device according to claim 1 ,
comprising a microphone matcher configured to perform a matching of the N frequency domain audio signals, resulting in N matched frequency domain audio signals.
8. The audio encoding device according to claim 7 , comprising
a diffuse sound estimator configured to estimate a diffuse sound power, and
a de-correlation filter bank configured to perform a de-correlation of the diffuse sound power by generating three orthogonal diffuse sound components from the diffuse sound estimate power.
9. The audio encoding device according to claim 8 ,
wherein the diffuse sound estimator is configured to estimate the diffuse sound power according to:
A
=
1
-
Φ
diff
2
,
B
=
2
Φ
diff
E
{
X
1
X
2
*
}
-
E
{
X
1
X
1
*
}
-
E
{
X
2
X
2
*
}
,
C
=
E
{
X
1
X
1
*
}
E
{
X
2
X
2
*
}
-
E
{
X
1
X
2
*
}
2
,
and
P
diff
[
k
,
i
]
=
-
B
-
B
2
-
4
AC
2
A
,
wherein:
P diff is the diffuse sound power,
E{ } is an expectation value,
Φ 2 diff is a normalized cross-correlation coefficient between N 1 and N 2 ,
N 1 is diffuse sound in a first channel, and
N 2 is diffuse sound in a second channel.
10. The audio encoding device according to claim 9 ,
wherein the de-correlation filter bank is configured to perform the de-correlation of the diffuse sound power by generating three orthogonal diffuse sound components from the diffuse sound estimate power:
{tilde over (D)} W [ k,i ]=DFR W w u U 1 P 2D-diff [ k,i ],
{tilde over (D)} X [ k,i ]=DFR X w u U 2 P 2D-diff [ k,i ], and
{tilde over (D)} Y [ k,i ]=DFR Y w u U 3 P 2D-diff [ k,i ],
wherein:
DFR
a
=
Δ
1
4
π
∫
-
π
2
π
2
∫
-
π
π
R
a
(
θ
,
ϕ
)
2
cos
ϕ
d
θ
d
ϕ
,
R
X
(
θ
,
ϕ
)
=
cos
ϕ
cos
θ
,
R
Y
(
θ
,
ϕ
)
=
cos
ϕ
sin
θ
,
R
W
(
θ
,
ϕ
)
=
1
,
and
w
u
[
n
]
=
exp
(
-
0.5
ln
1
e
6
n
f
s
RT
60
)
with
-
l
u
<
n
<
l
u
,
wherein {tilde over (D)} W [k,i] is a first channel diffuse sound component,
wherein {tilde over (D)} X [k,i] is second channel diffuse sound component,
wherein {tilde over (D)} Y [k,i] is third channel diffuse sound component,
DFR W is a diffuse-field response of the first channel,
DFR X is a diffuse-field response of the second channel,
DFR Y is a diffuse-field response of the third channel,
w u is an exponential window,
RT 60 is a reverberation time,
U 1 ,U 2 ,U 3 is the de-correlation filter bank,
u is a Gaussian noise sequence,
l u is a given length of the Gaussian noise sequence, and
P 2D-diff is the diffuse noise power.
11. The audio encoding device according to claim 1 ,
comprising an adder, which is configured to add channel-wise, the first-order ambisonic B-format direct sound signals and the higher order ambisonic B-format direct sound signals, and/or the diffuse sound signals, resulting in complete ambisonic B-format signals.
12. The audio encoding device according to claim 1 ,
wherein delay estimator configured to estimate the angle of incidence for each pair of the N audio signal based on a travelling time delay between the pair of audio signals.
13. The audio encoding device according to claim 1 ,
wherein delay estimator configured to estimate the angle of incidence for each pair of the N audio signal based on a delay in second and a delay in samples between the pair of audio signals.
14. An audio recording device comprising the N microphones configured to record the N audio signals, and the audio encoding device according to claim 1 .
15. A method for encoding N audio signals, from N microphones where N≤3, the method comprising:
estimating angles of incidence of direct sound by estimating for each pair of the N audio signals an angle of incidence of the direct sound,
deriving A-format direct sound signals from the estimated angles of incidence by deriving, from each of the estimated angles of incidence, a respective one of the A-format direct sound signals, each of the A-format direct sound signals being a first-order virtual microphone signal, and
encoding the A-format direct sound signals in first-order ambisonic B-format direct sound signals by applying a transformation matrix to the A-format direct sound signals,
wherein N=3,
wherein the encoding further comprises performing a short time Fourier transformation on each of the N audio signals x 1 , x 2 , x 3 , resulting in N short time Fourier transformed audio signals X 1 [k,j], X 2 [k,j], X 3 [k,j],
wherein the method further comprises:
determining cross spectra of each pair of the short time Fourier transformed audio signals according to:
X 12 [k,i]=α X X 1 [k,i]X 2 * [k,i]+ (1−α X ) X 12 [k− 1, i],
X 13 [k,i]=α X X 1 [k,i]X 3 * [k,i]+ (1−α X ) X 13 [k− 1 ,i], and
X 23 [k,i]=α X X 2 [k,i]X 3 * [k,i ]+(1−α X ) X 23 [k− 1 ,i],
determining an angle of the complex cross spectrum of each pair of the short time Fourier transformed audio signals according to:
ψ
~
12
[
k
,
i
]
=
arctan
j
X
12
[
k
,
i
]
X
12
*
[
k
,
i
]
X
12
[
k
,
i
]
+
X
12
*
[
k
,
i
]
,
ψ
~
13
[
k
,
i
]
=
arctan
j
X
13
[
k
,
i
]
X
13
*
[
k
,
i
]
X
13
[
k
,
i
]
+
X
13
*
[
k
,
i
]
,
and
ψ
~
23
[
k
,
i
]
=
arctan
j
X
23
[
k
,
i
]
X
23
*
[
k
,
i
]
X
23
[
k
,
i
]
+
X
23
*
[
k
,
i
]
,
performing a phase unwrapping to {tilde over (ψ)} 12 {tilde over (ψ)} 13 {tilde over (ψ)} 23 , resulting in ψ 12 ψ 13 ψ 23
estimating the delay in number of samples according to:
δ 12 [ k,i ]=( N STFT /2+1)/( i π)ψ 12 [ k,i ],
δ 13 [ k,i ]=( N STFT /2+1)/( i π)ψ 13 [ k,i ],
δ 23 [ k,i ]=( N STFT /2+1)/( i π)ψ 23 [ k,i ], if i≤i alias
or
δ 12 [ k,i ]=( N STFT /2+1)/( i π)Ψ 12 [ k,i ],
δ 13 [ k,i ]=( N STFT /2+1)/( i π)Ψ 13 [ k,i ],
δ 23 [ k,i ]=( N STFT /2+1)/( i π)Ψ 23 [ k,i ], if i>i alias
estimating the delay in seconds according to:
τ
12
[
k
,
i
]
=
δ
12
[
k
,
i
]
f
s
,
τ
13
[
k
,
i
]
=
δ
13
[
k
,
i
]
f
s
,
and
τ
23
[
k
,
i
]
=
δ
23
[
k
,
i
]
f
s
and
estimating the angles of incidence according to:
θ
12
[
k
,
i
]
=
arcsin
(
c
τ
12
[
k
,
i
]
d
m
i
c
)
,
θ
13
[
k
,
i
]
=
arcsin
(
c
τ
13
[
k
,
i
]
d
m
i
c
)
,
and
θ
23
[
k
,
i
]
=
arcsin
(
c
τ
23
[
k
,
i
]
d
m
i
c
)
,
and
wherein:
x 1 is a first audio signal of the N audio signals,
x 2 is a second audio signal of the N audio signals,
x 3 is a third audio signal of the N audio signals,
X 1 is a first short time Fourier transformed audio signal of the short time Fourier transformed audio signals,
X 2 is a second short time Fourier transformed audio signal of the short time Fourier transformed audio signals,
X 3 is a third short time Fourier transformed audio signal of the short time Fourer transformed audio signals,
k is a frame of the short time Fourier transformed audio signals, and
i is a frequency bin of the short time Fourier transformed audio signals,
X 12 is a cross spectrum of a pair of X 1 and X 2 ,
X 13 is a cross spectrum of a pair of X 1 and X 3 ,
X 23 is a cross spectrum of a pair of X 2 and X 3 ,
α x is a forgetting factor,
X* is a conjugate complex of X,
j is the imaginary unit,
ψ 12 is an angle of the complex cross spectrum of X 12 ,
ψ 13 is an angle of the complex cross spectrum of X 13 ,
ψ 23 is an angle of the complex cross spectrum of X 23 ,
i alias is a frequency bin corresponding to an aliasing frequency,
f s is a sampling frequency,
d mic is a distance of the microphones, and
c is the speed of sound.
16. A non-transitory computer readable storage medium comprising a computer program with a program code, which is configured to be executed by a computer to cause the computer to perform the method according to claim 15 .