Method and device for sound source localization
View Patent ↗A method and an apparatus for locating a sound source are provided. The method includes: obtaining M channels of audio signals of a preset format by microphone arrays located in different planes (S 100 ); preprocessing the M channels of audio signals of the preset format, and projecting them onto the same plane, so as to obtain N channels of audio signals, where M≥N (S 200 ); performing a time-frequency transformation on each of the N channels of audio signals, so as to obtain frequency domain signals of the N channels of audio signals (S 300 ); further calculating a covariance matrix of the frequency domain signals and performing a smoothing process (S 400 ); performing an eigenvalue decomposition of the smoothed covariance matrix (S 500 ); estimating the sound source direction according to an eigenvector corresponding to the maximum eigenvalue, so as to obtain a sound source orientation parameter (S 600 ).
1. A method for locating a sound source comprising:
step 1: obtaining M channels of audio signals of a preset format by using microphone arrays located on different planes, wherein M is a positive integer;
step 2: preprocessing the M channels of audio signals of the preset format, and projecting the M channels of audio signals of the preset format onto a same plane to obtain N channels of audio signals, wherein N is a positive integer, and M≥N;
step 3: performing a time-frequency transform on each of the N channels of audio signals to obtain frequency domain signals of the N channels of audio signals;
step 4: calculating covariance matrices of the frequency domain signals, and performing a smoothing process on each of the covariance matrices to obtain smoothed covariance matrices;
step 5: performing an eigenvalue decomposition on each of the smoothed covariance matrices to obtain N eigenvalues and corresponding eigenvectors; and
step 6: estimating a direction of the sound source according to an eigenvector corresponding to a maximum eigenvalue of the N eigenvalues, to obtain sound source orientation parameters,
wherein in the step 1, M=4, the preset format is an Ambisonic A format, and the four channels of audio signals (LFU, RFD, LBD, RBU) are located on different planes,
wherein a specific process of the preprocessing in the step 2 is:
converting he four channels of audio signals of the Ambisonic A format into three (N=3) channels of audio signals (L, R, S) in the same plane by a conversion matrix A:
[
L
R
S
]
=
A
×
[
LFU
RFD
LBD
RBU
]
wherein the conversion matrix
A
=
[
a
11
a
12
a
13
a
14
a
21
a
22
a
23
a
24
a
31
a
32
a
33
a
34
]
,
and values of elements a 11 , a 12 , . . . , a 34 of the conversion matrix A are constants and are determined by different sound source scenes.
2. The method for locating the sound source according to claim 1 , wherein a process of the preprocessing in the step 2 is:
converting the four channels of audio signals of the Ambisonic A format into four (N=4) channels of audio signals (F, R, B, L) in the same plane by the conversion matrix A:
[
F
R
B
L
]
=
A
×
[
LFU
RFD
LBD
RBU
]
wherein the conversion matrix
A
=
[
f
11
(
ϕ
)
f
12
(
ϕ
)
f
13
(
ϕ
)
f
14
(
ϕ
)
f
21
(
ϕ
)
f
22
(
ϕ
)
f
23
(
ϕ
)
f
24
(
ϕ
)
f
31
(
ϕ
)
f
32
(
ϕ
)
f
33
(
ϕ
)
f
34
(
ϕ
)
f
41
(
ϕ
)
f
42
(
ϕ
)
f
43
(
ϕ
)
f
44
(
ϕ
)
]
,
ϕ is a height angle, and ƒ(ϕ) is a function related to ϕ.
3. The method for locating the sound source according to claim 2 , wherein when the microphone arrays pick up the audio signals, if the sound source is in a middle position (ϕ=0°), the conversion matrix
A
=
[
a
11
a
12
a
13
a
14
a
21
a
22
a
23
a
24
a
31
a
32
a
33
a
34
a
41
a
42
a
43
a
44
]
,
and values of elements a 11 , a 12 , . . . , a 44 of the conversion matrix A are constants and are determined by different sound source scenes.
4. The method for locating the sound source according to claim 1 , wherein a process of the preprocessing in the step 2 is:
step 21: converting the four channels of audio signals of the Ambisonic A format into audio signals (W, X, Y, Z) of an Ambisonic B format by the conversion matrix A:
[
W
X
Y
Z
]
=
A
×
[
LFU
RFD
LBD
RBU
]
wherein the conversion matrix
A
=
[
a
11
a
12
a
13
a
14
a
21
a
22
a
23
a
24
a
31
a
32
a
33
a
34
a
41
a
42
a
43
a
44
]
,
and values of elements a 11 , a 12 , . . . , a 44 of the conversion matrix A are constants and are determined by different sound source scenes;
step 22: estimating a parameter of a divergence based on an energy of a Z signal in the audio signals of the Ambisonic B format;
step 23: determining whether the divergence is greater than a set threshold; and
step 24: if the divergence is greater than the set threshold, estimating the direction of the sound source by using three (N=3) channels of audio signals (L, R, S); and
if the divergence is not greater than the set threshold, estimating the direction of the sound source by using four (N=4) channels of audio signals (F, R, B, L).
5. The method for locating the sound source according to claim 1 , wherein the time-frequency transform in the step 3 is realized by a Discrete Fourier Transform (DFT), a Fast Fourier Transform (FFT) or a Modified Discrete Cosine Transform (MDCT).
6. The method for locating the sound source according to claim 1 , wherein a specific process of estimating the direction of the sound source in the step 6 is:
searching for, according to the eigenvector corresponding to the maximum eigenvalue of the N eigenvalues, an index value corresponding to a maximum inner product value by using an inner product of the eigenvector corresponding to the maximum eigenvalue of the N eigenvalues and a steering vector, wherein the index value corresponds to the direction of the sound source.
7. The method for locating the sound source according to claim 1 , wherein:
in the step 3, the frequency domain signals are divided into a plurality of sub-bands;
in the step 4, the covariance matrices are calculated for the plurality of sub-bands and the smoothing process is performed;
in the step 5, the eigenvalue decomposition is respectively performed on the covariance matrices of the plurality of sub-bands after the smoothing process to obtain N eigenvalues and corresponding eigenvectors of the covariance matrices of the plurality of sub-bands; and
in the step 6, the direction of the sound source is estimated for each sub-band of the plurality of sub-bands according to the eigenvector corresponding to the maximum eigenvalue, and the sound source orientation parameters are obtained in combination with detection results of the direction of the sound source for the each sub-band.
8. A device for locating a sound source, comprising: an acquisition unit of an audio signal of a preset format, a signal preprocessing unit, a time-frequency transform unit, a frequency domain signal processing unit, and a sound source orientation estimation unit, wherein
the acquisition unit of the audio signal of the preset format is configured to obtain M channels of audio signals of the preset format by using microphone arrays located on different planes, and send the M channels of audio signals of the preset format to the signal preprocessing unit, wherein M is a positive integer and M=4;
the signal preprocessing unit is configured to preprocess the M channels of audio signals of the preset format and project the M channels of audio signals of the preset format onto a same plane to obtain N channels of audio signals, and send the N channels of audio signals to the time-frequency transform unit, wherein N is a positive integer, and M≥N;
the time-frequency transform unit is configured to perform a time-frequency transform on each of the N channels of audio signals to obtain frequency domain signals of the N channels of audio signals;
the frequency domain signal processing unit is configured to process the frequency domain signals, calculate covariance matrices of the frequency domain signals and perform a smoothing process, further perform an eigenvalue decomposition on the covariance matrices to obtain eigenvalues and eigenvectors, and send the eigenvalues and eigenvectors to the sound source orientation estimation unit; and
the sound source orientation estimation unit is configured to estimate a direction of the sound source according to an eigenvector corresponding to a maximum eigenvalue of the eigenvalues, to obtain sound source orientation parameters,
wherein the preset format is an Ambisonic A format, and four channels of audio signals (LFU, RFD, LBD, RBU) are located on the different planes,
wherein the signal preprocessing unit is configured to
convert the four channels of audio signals of the Ambisonic A format into three (N=3) channels of audio signals (L, R, S) in the same plane by a conversion matrix A:
[
L
R
S
]
=
A
×
[
LFU
RFD
LBD
RBU
]
wherein the conversion matrix
A
=
[
a
11
a
12
a
13
a
14
a
21
a
22
a
23
a
24
a
31
a
32
a
33
a
34
]
,
and values of elements a 11 , a 12 , . . . , a 34 of the conversion matrix A are constants and are determined by different sound source scenes.